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Abstract— Merged characters are the major cause of recog- 
nition errors. We classify the merging relationship between two 
involved characters into three types: "linear," "nonlinear," and 
"overlapped." Most segmentation methods handle the first type 
well, however, their capabilities of handling the other two types are 
limited. The weakness of handling the nonlinear and overlapped 
types results from character segmentation by linear, usually 
vertical, cuts assumed in these methods. This paper proposes a 
novel merged character segmentation and recognition method 
based on forepart prediction, necessity-sufficiency matching and 
character-adaptive masking. This method utilizes the informa- 
tion obtained from the forepart of merged characters to predict 
candidates for the leftmost character, and then applies char- 
acter-adaptive masking and character recognition to verifying 
the prediction. Therefore, the arbitrary-shaped cutting path will 
follow the right shape of the leftmost character so as to preserve 
the shape of the next character. This method handles the first two 
types well and greatly improves the segmentation accuracy of the 
overlapped type. The experimental results and the performance 
comparisons with other methods demonstrate the effectiveness of 
the proposed method. 

Index Terms — Character-adaptive masking, character feature 
extraction, character segmentation, forepart prediction, merged 
character recognition, necessity-sufficiency matching. 



I. Introduction 

OPTICAL character recognition (OCR) is one of the most 
successful applications of automatic pattern recognition. 
Nowadays, the research interests of OCR focuses on the recog- 
nition of degraded, multiple-font machine-printed, and hand- 
written text [ 1], For printed text, the main difficulty comes from 
the severely merged or degraded characters. Until now, incor- 
rect recognition of merged characters is still one of the main 
causes for recognition errors. The well-known tests of commer- 
cial printed-text OCR systems by the University of Nevada, Las 
Vegas [2], show that even when perfect patterns were recog- 
nized, commercial systems experience 0.5% spacing errors in 
average. This is essentially a segmentation error by a process 
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that attempts to isolate word images [31. Furthermore, since seg- 
mentation errors often raise chain effects, the performance of 
segmentation is crucial for the whole OCR process. 

This paper proposes an effective merged-character recogni- 
tion method, which consists of a novel segmentation scheme 
based on forepart prediction (FP), character-adaptive masking 
(CAM), and a necessity-sufficiency matching (NSM) algorithm 
for single character recognition. First, the method predicts the 
candidates for the foremost (for horizontally aligned text, "fore- 
most" equals "leftmost") character of merged characters based 
on a set of forepart features and then cuts it from the image using 
its individual mask, namely, character-adaptive masking. Next, 
the recognizer determines the probability of each candidate and 
decides which candidates are acceptable, and each accepted can- 
didate will start a new processing branch for the latter segmen- 
tation; consequently, there will be multiple segmentation solu- 
tions. Finally, the solution with the highest holistic recognition 
probability will be accepted. Compared with existing merged- 
character recognition approaches, the advantages of the pro- 
posed method are the following: 

1) forepart prediction is verified by reliable character 
matching; 

2) NSM algorithm utilizes a concise feature-row-based char- 
acter model to distinguish similar characters efficiently; 

3) character masks for the segmentation are adaptive to char- 
acter shapes so as to avoid damaging the character image. 

The rest of this paper is organized as follows. Section II classi- 
fies the merging types and reviews the related work. Section III 
describes the proposed algorithms in detail. The experimental 
results and comparisons with other methods are reported in Sec- 
tion IV. Finally, Section V draws our conclusions. 

II. Related Work 

The recognition methods of merged characters can first 
be divided into two classes: one with segmentation, and one 
without. The methods without segmentation recognize char- 
acters from a text image directly. Most segmentation-free 
methods are lexicon-based. They treat the word as a single, 
indivisible entity, and attempt to recognize it using features 
of the word as a whole. Thus, they are usually called holistic 
methods. Madhvanath and Govindaraju [4] conduct a survey 
on holistic methods and summarize "their treatment of lexicon 
words as distinct pattern classes has traditionally limited their 
application to recognition scenarios involving small, static 
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lexicons." Rocha and Pavlidis [5] propose an interesting seg- 
mentation-free approach, which converts a text image into a 
feature graph and then attempts to match subgraphs of features 
with predefined character prototypes. Different alternatives are 
represented by a net whose nodes correspond to the matched 
subgraphs. A final search for the optimal path in the net gives 
the best interpretation of the text image. Martin et al. [6] 
engage in an input window to scan horizontally over the text 
image, and train a neural network to recognize whether the 
window is centered over a single character (if so, the character 
is recognized) or between characters. Lee and Kim [7] improve 
this idea by employing the cascade neural network to retain the 
spatial relationship of connected characters. 

Generally, most character recognition methods [8]-[23] in- 
clude a segmentation process, i.e., first segmenting a given text 
image into character images and then recognizing each char- 
acter image separately. OCR methods for the single character 
recognition have been well studied in the literature [24], The 
current directions to improve the OCR performance include 
designing better classifiers [25], [26], integrating several ex- 
isting classifiers to utilize various kinds of features [27], and 
extracting more robust features [28]-[30]. This paper proposes a 
new feature extraction and matching approach based on FP and 
NSM. Jung et al. [19] also explores the leftmost and rightmost 
features to recognize merged characters; however, they use linear 
segmentation, while the proposed method conducts nonlinear 
segmentation. 

Prior to discussing the segmentation approaches, we define 
the three types of merging relationships: "linear" "nonlinear," 
and "overlapped 1 (Fig. 1). In Fig. 1(a), the two merged charac- 
ters can be completely separated by a linear segmentation path, 
defined as the "linear" type. Here the "completely separated" 
means the entirety of both characters is maintained. In Fig. 1(b), 
the two merged characters cannot be completely separated by 
any linear path, but can be done by a nonlinear path, defined 
as the "nonlinear" type. In Fig. 1(c), the two merged charac- 
ters overlap each other; therefore, they cannot be completely 
separated by either a linear path or a nonlinear path, defined 
as the "overlapped" type. Following the definition of merging 
types, we classify a character segmentation approach into either 
a linear segmentation approach or a nonlinear segmentation ap- 
proach, according to the shape of its segmentation path. 

Linear segmentation approaches separate merged characters 
by linear cuts, mostly vertical cute. They depend on a variety of 
clues to locate the cutting positions. The most simple and fast 
way is to analyze the vertical projection of a black region. The 
representative methods are [8], [9], which locate the valleys of 
the projection profile as the cutting positions. However, they 
cannot handle the linear type with long vertical connected 
length [Fig. 2(a)], degraded characters, or italic characters. 
Tsujimoto and Asada [10] proposed a filter to produce a 
deeper valley at merging positions using an "and" operation 
of adjacent columns, but the improvement is limited. Another 
popular way is recognition-based segmentation, which gener- 
ates multiple segmentation hypotheses and then chooses the 
best one by recognition. The approach of Casey and Nagy [14] 
recognizes the subimage in a shrinking window whose width 
decreases leftward [Fig. 2(b)]. Once a character is matched, its 
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Fig. 1. Three merging types, (a) Linear, (b) Nonlinear, (c) Overlapped. 

corresponding block will be cut and the remainders will be pro- 
cessed recursively. This strategy is better than projection-based 
methods because it selects cutting positions dynamically, but 
it also cannot handle the nonlinear type due to the vertical 
cutting. Other recognition-based segmentation approaches 
[15], [19] propose different criteria to select cutting positions, 
however, they still perform vertical cutting. To decrease the 
segmentation errors caused by vertical cuts, Bayer et al. [11], 
[12] place a cut classifier before cutting, and perform addi- 
tional connectivity analysis after cutting to correct the errors. 
Some connected-component based approaches [13], [18], [22] 
employ a splitting-and-merging scheme. They first separate the 
text image into primitive blocks and then combine consecutive 
blocks into candidate characters using certain criteria, such 
as knowledge-based dynamic programming [18] and lexicon 
matching [22]. Recently, Garain and Chaudhuri [23] propose a 
new approach applying fuzzy multifactorial analysis to identify 
touching parts and segment merged characters by vertical cuts. 

Nonlinear segmentation approaches [16], [17], [20], [21] at- 
tempt to separate merged characters with various merging types 
by the optimal nonlinear cutting paths. Wang and Jean [ 1 6] pro- 
pose an approach called "shortest path segmentation." By pre- 
defining the "distance (cost)" needed for selecting the path of 
each hypothetic cut and then finding the "shortest" one, this 
method selects the optimal segmentation from all "legal" ones. 
This approach partially overcomes the problems raised by ver- 
tical cutting, and in a way, can adjust the path from vertical cut- 
ting. The shortest path consists of several vertical monochromic 
runs, as shown in Fig. 3. However, since the adjustment depends 
only on the local connectivity analysis, it cannot guarantee that 
the shortest path preserves the character's entirety well. Lee et 
al [17] extend the shortest path search to the grayscale image 
space. Arica and Yarman-Vural [21] define a novel cost function 
using the information extracted from both grayscale image and 
binary image, and employ a dynamic programming algorithm 
to search the shortest path. The above three approaches share 
the same concept, i.e., searching the shortest path based on the 
pixel- wise cost accumulation. Chang and Chen [31] introduce 
the convex-hull information to improve the accuracy of shortest 
path location. Chen and Wang [20] further propose a different 
approach which first performs thinning on both foreground and 
background regions for the feature points extraction on the fore- 
ground and background skeletons, then constructs several seg- 
mentation paths from these feature points, and finally decides 
the best segmentation path or rejects the segmentation by the 
mixture Gaussian probability function. 

In summary, existing nonlinear segmentation approaches 
all depend on local image features to determine segmentation 
paths, giving no guarantee to maintaining characters' entirety. 
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Fig. 2. Various segmentation methods, (a) Projection-based segmentation, (h) Recognition-based segmentation. 



Fig. 3. Shortest path segmentation. 

The proposed method differs from existing approaches in that 
the nonlinear segmentation path is formed by character-adaptive 
masking. We describe our approach in the next section in detail. 

m. Algorithms 

Correct recognition of merged characters doubtlessly relies 
on the accuracy of segmentation, which in turn needs image 
analysis and recognition to avoid the blindness of segmentation. 
We believe that the segmentation integrating both image anal- 
ysis and recognition is the most promising way. Considering 
a horizontal string of merged characters, its leftmost part, i.e., 
the forepart of the first character, keeps the original shape. 
Thus, if we can 1) predict a group of candidates by analyzing 
the forepart, 2) verify the candidates by character recognition, 
and finally 3) cut out the subimage of the recognized character 
without damaging the succeeding character, then the entire 
string of merged characters can be recognized recursively. This 
is the fundamental idea of our proposed method. 

The three conditions reveal the three key techniques of our 
method: FP-based prediction, NSM-based character recognition, 
and character-adaptive masking. This section will explain our 
method in a bottom-up order. Since the character recognition is 
an elementary process, it will be described first. The prediction 
and masking algorithms are presented next. Finally, we illustrate 
the overall paradigm of the merged characters recognition. 

A. Character Recognition 

Being an elementary process, the character recognition will 
be invoked very frequently; therefore, it should be time-ef- 
ficient and accurate. The traditional character bitmap model 
cannot satisfy this requirement. Letting the prototype binary 
bitmap of a character be M and the input binary bitmap of a 
character be X, the resulting binary bitmap of matching M 
and X is calculated by R = M 8 X, The nonzero points in R 
are called different points (DP). A DP caused by degradation 
or displacement of the same stroke is called Edge_DP, as 
shown in Fig. 4(a). A DP caused by different strokes is called 



Fig. 4. Examples of DPs. (a) Edge_DP M = :t O;" X = "0;" (b) Stroke_DP 
M = "0;" X = "U." 



StrokeJDP, as shown in Fig. 4(b). Since the character recogni- 
tion is normally to detect Stroke_DPs and tolerate Edge_DPs, 
judging only by the quantity of DP is not robust for degradation 
and displacement; for example, Fig, 4(a) and Fig. 4(b) contain 
a similar number of DPs. On the other hand, distinguishing 
Edge_DP from Stroke_DP requires more complex global anal- 
ysis. Therefore, the entire bitmap model is not a good choice. 
Moreover, the entire bitmap model contains much redundancy 
whose contribution to the recognition is less than the noise it 
introduces. Thus, we design a new compact feature extraction 
and matching algorithm. 

1) Feature matching algorithm: To reduce the processing 
time and to achieve the maximum marginal utility of character 
features, the number of features should be the minimal that 
can distinguish English characters and punctuations. Since the 
bitmap data is organized row by row, computation in the row 
direction is fast. Therefore, we define the character feature as 
a group of rows in the prototype bitmap of a character, called 
feature rows, denoted by /. 

Theoretically, the feature matching function should check 
whether each feature row and its corresponding row in an input 
bitmap are identical, as described in Definitions 1 and 2. 

Definition 1: Let MATCH(/, X) be the matching function 
of / and an input bitmap X. If they match, the function returns 
1; otherwise, it returns 0, 

Definition 2: Let Mi be a row in the prototype bitmap M of 
a character, where i is the row index. Let M' be the prototype 
bitmap of another character. A group of rows of M, denoted by 
/ m = {Mi\i = n,r 2 ... r k ], can be the feature rows of this 
character on the condition that 



MATCH(/ m , M ') = 



J 1, if and only if M — M' 
\0, M^M'. 



(1) 



However, since the scanned image often contains degradation 
or displacement, the condition in (1) is too strict. Consequently, 
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we propose the necessity-sufficiency matching algorithm based 
on dilated feature rows and contracted feature rows. 

Definition 3: Let f t (t = 1 . . . k) be a feature row consisting 
of several consecutive black and white runs. Then, the dilated 
feature row of f t , denoted by ff, is formed by adding two more 
black points to both the head and the tail of each run. Similarly, 
the contracted feature row of f u denoted by / t c , is formed by 
removing two head points and two tail points from each run; 
however, each run has to retain at least one point. Thus, / = 
{(fuft,f t d )\t= l.-fc}. 

With Definition 3, if X resembles M, X should cover ft(t = 
1 . . . k) completely and be covered by ff(t - 1 ... k) com- 
pletely. Suppose that for each character, its prototype bitmap and 
the input bitmap have been normalized to the same width (W) 
and height (H). The necessity matching condition is defined as 
(2), where row{t) returns the row index of t in the bitmap 

w^) = E^ J ) = 0 

w 

where N t {f u X) = £ [(/£ A X Iow{t)i ) © . (2) 
And, the sufficiency matching condition is defined as (3) 

s(f,x) = *£s t (f t ,x) = o 

w 

where S t (f t , X) = £ [(x row(t)j A /£) © X row(t)j ] . (3) 

Thus, we can define the necessity-sufficiency matching func- 
tion as follows: 

fN(f,X) = 0 2ndS(f,X) = 0 
otherwise. 

(4) 

Equation (4) judges whether X matches the feature rows of 
a character. The binary output is adequate for selecting feature 
rows based on Definition 2, but it cannot indicate to which ex- 
tent X resembles the character. Considering this information is 
important for the segmentation to make decision when multiple 
solutions exist, we modify the above equations to yield the prob- 
ability of X being the specific character. Equations (2)-(4) are 
modified to be (5)-(7), respectively. 

N'(f,X) = Y[[l-N^f t ,X)] l where 

Wt-*)=^-E [(/£■ A *«-*w)©tf] (5) 

S'(f,X) = H[l~S' t (f t ,X)}, where 

S '^ X )=W Mi A /w)©*r«(t M ] (6) 

MATCH'(/,X)=jV'(/,X) x S'{f,Xl 

0 < N'(f,X) < 1 and 
0<S\f,X)<l. (7) 



MATCH (/, 



Thus, (2)-(4) form the NSM algorithm for selecting feature 
rows, whereas (5)-(7) form the NSM algorithm for character 
recognition. An input bitmap X can be recognized as the char- 
acter with the maximum matching probability. 

The normal, dilated and contracted feature rows can tolerate 
horizontal displacement, but they are font-dependent. This de- 
sign is based on the consideration that it will enhance the capa- 
bility of distinguishing similar characters. 

2) Selection of feature rows: The process of selecting fea- 
ture rows for a character consists of two steps 1) preparing the 
prototype bitmap for this character, and 2) automatically gener- 
ating candidate feature. 

The feature rows are independent from font size since both 
the prototype bitmap and the input bitmap are normalized. Since 
they are dependent on font style, we generate separate feature 
libraries for ten commonly-used fonts. This fashion is suitable 
for our application — tenders, which require specific fonts in spe- 
cific parts, so we can preload the feature library of the specific 
font by tender layout analysis. In each feature library, the pro- 
totype bitmap of every character has three variants for regular, 
bold, and italic styles, respectively. 

With the prototype bitmaps of all characters ready, the fea- 
ture rows of each character are automatically selected by the 
following four steps. For the current character, denote the pro- 
totype bitmap by M, the target feature row set by / r , and the 
candidate row set by C r . 

1) Initially, f r = {}, and C r contains all rows in M ; 

2) For each row r in C r , first make f' r {r) = f r + {r}, then 
count the times of f' r matching other prototype bitmaps 
than M, denoted by W , in the same feature library: 
M_SUM(r) = EMATCH(f t '(r), M'), finally select the 
r min with the minimum Af-SUM(r). If more than one r 
reach a tie for the r min , select the one with the maximum 
vertical distance to the existing feature rows in f r to 
distribute the feature rows evenly; 

3) Move r mm from C T to f r ; 

4) If MSUM{r min ) = 0, the current f r is the feature row 
set of M, then terminate. Otherwise, go to (2). 

The automatic feature row selection is offline and one-off; 
therefore, the time efficiency is not critical. The number of fea- 
ture rows for each character is different. According to our ex- 
periment, the number is at most five. Most characters have four 
or five feature rows. 

Fig, 5 shows a tool for monitoring the feature rows of a se- 
lected character. The left side shows the prototype bitmap and 
the feature rows. The current feature row together with its con- 
tracted and dilated feature rows is drawn at the bottom. The right 
side displays the matched characters of each feature row, and the 
box "matched chars" gives the characters matching ali feature 
rows. Only the ground-truth character should appear in this box. 

An important issue before matching feature rows is the 
alignment between the input bitmap and the feature rows, 
especially the vertical alignment since the NSM algorithm can 
tolerate small horizontal offsets. There are two common align- 
ment methods: bounding-box alignment and gravity-center 
alignment. The former is simple but sensitive to noise, while 
the latter is antinoise but computation-expensive. We utilize the 
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Fig. 5. Feature rows. 

baseline of printed text line (Fig. 6) for alignment. The baseline 
is defined at the bottom of upper case characters. Since the 
baseline is obtained from a text line, it is more antinoise than 
the gravity center, and it is fast. Therefore, the baseline position 
of each character is also labeled in the feature library. 

B. FP-based segmentation 

The FP-based segmentation algorithm consists of two steps: 
forepart prediction and character-adaptive masking. The former 
selects a group of candidates by forepart analysis, while the 
latter screens out the bitmap using the nonrectangular mask 
adaptive to each candidate and then recognizes the segmented 
bitmap by the NSM algorithm. Therefore, the merged charac- 
ters can be segmented with the highest recognition probability. 

/ ) Forepart prediction: After horizontally scanning a docu- 
ment image, we obtain the baseline position and the height of 
each text line, which is the vertical distance between the base- 
line and the top of the text line. Denoting the height of text line 
by Hi, the term "forepart" means the leftmost Hi /4 wide part of 
the input image of merged characters. For a prototype bitmap, 
it is nearly the left half of the bitmap. The forepart predication 
is based on three reliable forepart features, i.e., baseline-related 
feature, forepart height feature, and forepart boundary feature. 

The baseline-related feature indicates whether the forepart 
of a character has components under the baseline, which takes 
value "true" or "false." The forepart height feature indicates 
whether the forepart occupies the full height above the baseline, 
which also takes value "true" or "false." The forepart boundary 
feature is a little more complex, defined as follows: 

B = {B(i)\i is the row index of a bitmap 

and baseline < i < baseline + H{\ . (8) 

B[i) is the number of white points before the first black 
point in the row i. Considering the input scanned bitmap 
may contain noises, it can be revised to be the number of 
points before the first black segment whose width is not 

31 



Fig. 6. Baseline of a text line. 

less than the minimum stroke width. The difference between 
two forepart boundary features £?i and B 2 is defined as 
\B l -B 2 \=^ ,. MAX \\Bi{i)-B*(i)\}. 

baseline < i<baselme+Hi 

These three features of each character are obtained from its 
prototype bitmap and then stored in the feature library. Denote 
the baseline-related feature by L, the forepart height feature 
by G, and the forepart boundary feature by B. Correspond- 
ingly, these three features obtained from the forepart of an input 
bitmap are denoted by V , G 1 and B', respectively. Letting the 
initial candidate set Co containing all models in a feature library, 
we can select candidates by the following operations: 

Ci ={ce c Q \L = L'} 

C 2 = {c g Ci\G = G'} 

C 3 = {ceC 2 \\B-B'\<e}. 

The dimension of C 3 is much smaller than that of C" 0 , how- 
ever, candidates like "P" and "K" cannot be distinguished. To 
further reduce the number of candidates, for each candidate in 
we perform a quick necessity matching by (2) to form C 4l 
which is the final candidate set produced by the forepart predic- 
tion. 

C i = {ceC s \N(f c ,X)^0}. 

2) Character-adaptive masking: The forepart prediction 
produces a small number of candidates. The best way to verify 
each candidate is to screen out the bitmap of the leftmost char- 
acter and recognize it. To avoid the disadvantages of vertical 
cutting, we propose a character-adaptive masking algorithm, 
which uses an individual mask for each character. The mask is 
generated from the prototype bitmap of the character. Before 
defining the mask, we first define the right boundary of the 
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character. Recall the width and height of the prototype bitmap 
are W and H , respectively. The right boundary is defined as 
follows: 

RB = {RB(i)\ i is the row index of 

a bitmap, and 0 < i < H} 

where RB(i) is the horizontal coordinate of the rightmost black 
point on the row i. If the entire row is white, RB(i) = 0. Then, 
the real width of the character W c = MAX[RB(i)]. Now we 
can define the character mask. 

Definition 4: The mask is a binary matrix whose row index 
is the same as that of the prototype bitmap and whose width is 
W c . The value of each point in the mask is defined by (9), also 
as shown in Fig. 7. 

M-k(i.i) = (i' 0<i<MAX(i^),^) (9) 
v (0, otherwise. 

Notice that the right bound "MAX(RB(i),W c /2)" is impor- 
tant for the sufficiency matching. For example, when the left- 
most character is "B," the candidates may include "B," "P," 
an "L." If simply using RB(i) as the right bound, the masked 
bitmap of "P" or "L" will also be recognized perfectly. Using 
the mask of a candidate (denoted by c) to "and" with the input 
bitmap pixel by pixel, we can segment the bitmap of the candi- 
date (X c ) from the remainder along the path determined by the 
right shape of the candidate. Character-adaptive masking is very 
effective in segmenting linear and nonlinear merging types. For 
the overlapped type, at least the shape of the left character can 
be preserved. 

C. Merged Character Recognition Paradigm 

Compute the recognition probability of c by (7), i.e., P c = 
MATCH'(/ C , X c ). If only accepting the candidate with the 
highest P c , the segmentation will have one solution; however, it 
also has a higher risk of rejection or misrecongition of the en- 
tire string of merged characters. On the other hand, if accepting 
every candidate whose P c is high enough, the segmentation may 
yield multiple solutions, and the best solution can be selected by 
the holistic recognition probability. For the best performance, 
we choose the latter way. 

The recognition of a scanned document image first separates 
the image into several areas by layout analysis, then divides each 
area into subimages of text lines by horizontal projection, and 
finally cuts each text line into character bitmaps at zero-height 
positions of vertical projection profile. The final bitmap may 
contain either single character or merged characters, which are 
distinguished by checking the aspect ratio and the side profile 
[19]. The bitmap of single character is recognized by the NSM 
algorithm directly, while the bitmap of merged characters be- 
comes the input bitmap X of our recognition paradigm shown 
in Fig. 8. 

Since this paradigm allows multiple solutions, we need to use 
a stack to store the branchpoint status for the recursive process. 
The structure of stack element consists of three variables, as 
shown in the double-bordered box at the top-left corner of Fig. 8. 
P h is the holistic recognition probability up to the branchpoint. 

3* 




Fig. 7. Mask of "P." 

"String" contains a list of recognized characters. "Bitmap" is 
the current remainder of bitmap to be recognized. Initially, the 
feature library is ready, and the stack contains only one element, 
whose P h is 1, "String" is empty, and "Bitmap" is the input 
bitmap of merged characters. 

The recursive process begins with popping up the top ele- 
ment of the stack. Then, the forepart prediction takes place to 
select a small number of candidates into C 4 by four proce- 
dures. Next, for each candidate c in C 4 , (1) the character-adap- 
tive masking screens out the bitmap of this candidate, denoted 
by X c , correctly. (2) The NSM-based character recognition is 
performed to obtain the recognition probability P c . (3) Check 
whether P c > £, where £ is a predefined threshold for accepting 
the recognition probability. If P c < £, this candidate is dis- 
carded and the following steps are skipped. (4) Check whether 
the remainder of the input bitmap, i.e., "bitmap"- X c , is empty. 
If so, all merged characters are processed. Therefore, a solu- 
tion, consisting of two variables: Pu and "String," is added to 
the solution array and the following step is skipped. (5) Push the 
current recognition status into the stack. The value of the stack 
element representing the current recognition status is assigned 
as follows: P h = P h x P Cl "string" is appended with the cur- 
rent candidate c, and "bitmap" is the remainder after masking. 
Once all candidates in C 4 have been processed, check whether 
the stack is empty. If so, the recognition finishes. Otherwise, 
perform the recursive process again. 

After the. recursive process finishes, we get n solutions. If 
n = 0, the rejection happens. If n = 1, the only solution 
becomes the final solution. Otherwise, there exist multiple so- 
lutions. Usually, the one with the highest holistic recognition 
probability (Ph) will become the final solution. When the vo- 
cabulary of the application is limited, spelling check is also 
useful. 

IV. Performance Analysis and Experiments 
A. Performance Analysis 

Table I presents a performance comparison on segmentation 
between the proposed method and other representative methods 
in terms of general characteristics, segmentation accuracy, 
adaptability to three merging types, and speed. The evaluation 
is based on the experimental results of implementing the pro- 
jection-based segmentation approach [8], recognition-based 
segmentation approach [14], and shortest-path segmentation 
approach [16]. 
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Table I shows that the segmentation quality of our proposed 
method is the best among the four methods. The advantages 
of our method come from the forepart prediction and the char- 
acter-adaptive masking. For the merged characters of the linear 
and nonlinear types, our method can segment them correctly, 
for those of the overlapped type, our method can correctly seg- 
ment at least the left character, while the other methods cannot 
guarantee segmenting any of them correctly. 

B. Experiments 

We implemented the proposed method in an experimental 
system — VHTender, which processes scanned tenders for con- 
struction companies. To evaluate the performance of the pro- 
posed algorithms in detail, we divide the experiment into two 
parts: NSM-based character recognition and FP-based merged 
character segmentation. 

I) Result of character recognition: The proposed 
NSM-based character recognition algorithm is compared 
with a commercial OCR system— OmniPage Pro 10 (OPP10). 
We choose OPP10 for two reasons: 1) OPP is a well-known 
OCR software with a long development history. Its developer, 



ScanSoft company, claims that OPP10 is one of the best OCR 
system in the world, 2) The testing documents in our experi- 
ment are construction tenders, which contain a lot of underlined 
texts. Many OCR systems cannot handle underlined texts well, 
especially when text touches its underline. However, OPP 10 is 
not very sensitive to it. 

The testing data are the scanned images of 12 tender pages, 
containing 12786 characters. Table II shows the recognition re- 
sults of two systems obtained in a PC (P1U500/256M). 

The number of errors includes both false recognition errors 
and rejection errors. We analyze the cause of each error and 
report the error distributions of two systems in Table IE. 

The experimental results show that both systems achieve high 
recognition rates. They have similar performances on degraded 
characters and merged characters; however, VHTender has a 
much better ability in distinguishing characters that are similar 
in shape, e.g., "1-1-1," which makes VHTender achieve a higher 
recognition rate. This advantage of VHTender comes from that 
the feature rows of the similar characters are automatically de- 
fined at the location of the maximum difference and the NSM 
algorithm is sensitive to the difference of even one feature row. 
The overall processing time (T_all) of VHTender is longer than 
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that of OPP10. Since VHTender is a system designed for auto- 
matic tender analysis, T_all includes the time for layout analysis 
and tender understanding. The pure time for OCR (T_OCR) is 
only one fifth of T_all. 

2) Result of character segmentation: We select another 20 
pages containing heavily merged characters to test the FP-based 
segmentation algorithm. Fig. 9 shows the nature of the testing 
image. The total number of merged characters and the distribu- 
tion of merging types are given in Table TV. 

Our FP-based segmentation algorithm (abbreviated to FPS) 
is compared with the shortest path segmentation algorithm [16] 
(abbreviated to SPS). For easy understanding, we describe the 
SPS algorithm briefly. A candidate cutting path Pi starts from 
each column in the bottom (or top) row of an input bitmap, then 
selects the least cost for P; to move to the next row upward (or 
downward), and finally stops when reaching the top (or bottom) 
row. At any row, Pi can only move to one of its left-diagonal, 
vertical, or right-diagonal neighbors in the next row. The cost for 
a move is defined as follows: a vertical move to a white pixel 
costs 0, a vertical move to a black pixel costs 10, a diagonal 
move to a white pixel costs 1, and a diagonal move to a black 
pixel costs 10V2. The SPS algorithm uses a one-dimensional 
(ID) cost array {d} to record the accumulative cost for each 
Pi and a two-dimensional (2-D) path array to store the route of 
each Pi. After obtaining the costs of all candidate paths, the path 
with the least cost, i.e., the shortest path, will be considered as 
the cutting path. 

Some examples of the segmentation results of the two algo- 
rithms are shown in Fig. 10, where the cutting paths are drawn 
in gray. The cutting path of the SPS algorithm consists of 8-con- 
nected vertical lines, whereas that of the FPS algorithm fol- 
lows the right boundary of the character mask. The left four 
examples are all with the Linear merging type and the ver- 
tical length of connected parts are very short. Both two algo- 
rithms segment them well with small difference, such as the 
bottom-left corner of "a" in the example "na," which demon- 
strates the FPS algorithm can preserve the shape of character 
better than the SPS algorithm. For the nonlinear merging type, 
e.g. "ob" and "on" in the right examples, the FPS algorithm does 
a better clean cut than the SPS algorithm. The abilities of two 
algorithms differs much in handling the overlapped merging 
types, in the example "omp," "om" is with the nonlinear type 
and "mp" is with the overlapped type. Since the lengths of con- 

3* 



complete the panel wall part 
alternatives in accordance w 
to obviate movement and any 
joints/junctions where suria 

Fig. 9. Fraction of testing image. 

nected parts are long, the cost of cutting through the black areas 
becomes very high. Consequently, the SPS algorithm mis-seg- 
ments "omp" into "cnp." On the contrary, the FPS algorithm 
segments three characters correctly by using forepart prediction 
and character-adaptive masking. The only imperfection of the 
FPS result is that "p" loses those pixels overlapping with "m." 

The analysis on the least cost profile of SPS (Fig. 11) reveals 
the three following theoretical disadvantages of the SPS algo- 
rithm: 

1) definition of cost prevents the cutting path from passing 
many black pixels, which, however, is a must to segment 
long connected parts; 

2) usually, the shortest path is not the best cutting path. 
Sometimes, many paths have similar costs. Therefore, 
the SPS algorithm has to invoke the character recognition 
to select the best path, which somewhat diminishes the 
advantage of "shortest path.;" 

3) least cost profiles generated by bottom-up accumulation 
and top-down accumulation are not consistent. 

For example, the top-down cost profile provides a better path to 
segment "ve" rather than the bottom-up profile. However, the 
SPS algorithm did not consider this inconsistence. Compared to 
the "shortest path" concept, the segmentation based on forepart 
prediction is more straightforward and goal-directed. 

Table IV reports the quantitative experimental results, which 
demonstrates that the FP-based segmentation achieves remark- 
able improvement when handling the nonlinear and overlapped 
merging types. The improvement attributes to three factors: the 
effective forepart prediction, the character-adaptive masking 
and the accurate NSM-based character recognition. 

V. Conclusion 
This paper classifies the merging relationship between two 
involved characters into three types: linear, nonlinear, and 
overlapped. Existing segmentation methods usually handle 
the linear type well; however, their capabilities of handling 
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the other two types are rather limited. This paper proposes a 
novel merged-character recognition method based on forepart 
prediction, NSM-based recognition, and character-adaptive 
masking. This method handles both the linear and the non- 
linear merging types well. For the overlapped type, the left 
character is segmented correctly at the cost of possibly breaking 
the entirety of the right character. In summary, this paper makes 
two main contributions. The first one is the NSM-based 
character recognition. This algorithm utilizes feature rows to 
reduce the redundancy of character feature such that the feature 
rows are selected to maximize the difference among similar 
characters; therefore, this algorithm is fast and accurate. Due 
to the use of dilated and contracted feature rows, it is also 
antinoise and able to tolerate one- or two-pixel horizontal 
displacement between the character model and the character 
bitmap to be matched. The second contribution is the FP-based 
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segmentation using forepart prediction and character-adaptive 
masking. The effective forepart prediction quickly eliminates 
most irrelevant candidates, and the character-adaptive masking, 
making this segmentation algorithm the first one engaging 
character-dependent cutting path, is very important to preserve 
the shapes of merged characters. The experimental results and 
the performance comparisons with other methods demonstrate 
that the proposed method achieves remarkable improvement 
over existing methods. 
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Abstract— A semantic categorization method for generic home 
photo is proposed. The main contribution of this paper is to 
exploit a two-layered classification model incorporating camera 
metadata with low-level features for multilabel detection. The 
two-layered support vector machine (SVM) classifiers operate to 
detect local and global photo semantics in a feed-forward way. 
The first layer aims to predict likelihood of predefined local photo 
semantics based on camera metadata and regional low-level visual 
features. In the second layer, one or more global photo semantics 
is detected based on the likelihood. To construct classifiers pro- 
ducing a posterior probability, we use a parametric model to fit 
the output of SVM classifiers to posterior probability. A concept 
merging process based on a set of semantic-confidence maps is also 
presented to cope with selecting more likelihood photo semantics 
on spatially overlapping local regions. Experiment was performed 
with 3086 photos that come from MPEG-7 visual core experiment 
two official databases. Results showed that the proposed method 
would much better capture multiple semantic meanings of home 
photos, compared to other similar technologies. 

Index Terms— Camera metadata, image classification, photo 
album, support vector machine. 



I. Introduction 

THE GOAL of semantic image categorization is to dis- 
cover the image semantics from a domain of some given 
predefined concepts, such as building, waterside, landscape, 
cityscape, and so forth. Recently, as it is affordable to keep 
a complete digital record of one's whole life, the need for 
semantic categorization has been raised in both organizing 
and managing personal photo collection for minimizing user's 
manual efforts. 

Conventionally, many researches have advanced semantic 
image indexing and categorization in recent decades [l]-[9]. 
They mostly focused on reducing the semantic gap between 
low-level visual features and high-level semantic descrip- 
tions, which are closer to human visual perception. Herein, 
one primary tackling point is learning approach itself so that 
classifier realizes minimal bound of error in real application. 
In particular, statistical learning approaches, such as Bayesian 
probability model [32], Markov random fields (MRF) [1], and 
support vector machines (SVM) [2], have been successfully 
employed for semantic categorization. A statistical learning 
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process commonly includes three steps: 1) observing a phe- 
nomenon in the real world; 2) constructing a model of the 
phenomenon; and 3) making predictions using the model, step 
by step. A useful approach to build the model of a classifier 
is to employ discriminative features besides low-level visual 
features or any combination of both. J. Smith et al, in [7], has 
proposed semantic image/video indexing using semantic model 
vectors that are constructed from multiple low-level features 
where the model vector stands for a set of numerical degrees of 
strength in relation to different semantic meanings. For better 
classification, in [2] and [9], spatial image context features have 
been coupled with low-level features as well. 

Human beings sense many levels of visual semantics in photo. 
However, semantic labels to be discovered are generally lim- 
ited in a specific domain according to application, due to uncer- 
tainty and infinity of semantic knowledge of human beings. Al- 
though semantic object segmentation has been implemented by 
a wide range of approaches for last two decades [33]-[36], how 
to detect multiple semantic concepts in image is still challenging 
problem due to low performance and high computational cost. 

The problems in semantic categorization can be simplified by 
using multilayered rather than single-layered approach. Having 
multiple layers in classification often help to solve a classical 
image understanding problem that requires effective interaction 
of high-level semantics and low-level features. The way human 
beings perceive semantic knowledge of an image is hierarchical. 
In other words, human beings firstly sense rough, rather simple 
semantic objects, and then compound them to understand more 
comprehensively detailed semantic meanings of the image. This 
sensory mechanism can be imitated by a multilayered learning 
way. Multilayered approach usually forms a specific hierarchy 
of layers with one or more classifiers. A classifier in the lower 
layer aims to capture simple semantic aspects by using low-level 
features while a classifier in the higher layer interprets more 
complex semantic aspects by using high-level semantic features. 
Many researchers have employed the multilayered approaches 
to semantic categorization [5], [6], [9], [13]— [15], 

One state-of-the-art classification method is using SVM [10], 
[11], Many conventional classifiers have targeted empirical risk 
minimization (ERM). But, ERM only utilizes the loss function 
defined for a classifier and is equivalent to Bayesian decision 
theory with a particular choice of prior. Thus, an ERM approach 
often leads to an over-fitted classifier, i.e., classifier is usually 
too much adapted only to training data. Unlike ERM, structural 
risk minimization (SRM) minimizes generalization error. The 
generalization error is bounded by the sum of training set error 
and a term depending on Vapnik-Chervonenkis (VC) dimension 
of the learning machine. High generalization can be archived 
by minimizing the upper bound. SVM is based on the idea of 
SRM. The generalization error of SVM is related not to the input 
dimensionality of the problem, but to the margin with separating 
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the data. This explains why SVM can have a good performance 
even in problems with a large number of inputs. So far, SVM 
has been applied successfully to a wide range of problems. 

Unfortunately, naive SVM is inappropriate for multilayered 
classifier because the output of the SVM should be a calibrated 
posterior probability so as to enable post-processing. Basically, 
SVM is a discriminative classifier, not based on any generative 
model. Kernel function in the SVM acts as the only distance 
metric. For the use of SVM ensembles in multilayer, the output 
of all classifiers in a certain layer should be probabilistically 
modeled before being used as the input of all classifiers in the 
next layer. A few studies have tackled this problem [16], [17]. In 
[16], a parametric model was proposed to fit the SVM output to 
the posterior probability, instead of estimating class-conditional 
density. The parameters of the model are adapted to give the 
best probability output In [17], the parametric model studied in 
[16] has been improved for more optimized implementation. It 
solved the problem that the implementation of the conventional 
parametric model might not converge to the minimum solution. 
Although the new method increases complexity, it gives better 
convergence properties. 

Nevertheless, reliably capturing high-level visual semantics 
with low-level features remains a challenge to real application 
due to low performance and lack of generality. Unlike image, 
photo usually includes its camera metadata as well as pixel data 
itself. Camera metadata can be obtained from Exif header from 
photo file [1 8], In order to take best pictures in a given condition, 
a camera device is often adapted to surrounding light condition 
and subjects to be focused. Since camera metadata records this 
optimal picturing condition for a photo, it is of great benefit to 
semantic photo categorization in that it provides several useful 
cues that are independent on pixel data, thus giving some extra 
knowledge. At the early stage of utilization of camera metadata, 
taken date/time stamp has been successfully employed to cluster 
a sequence of unlabeled photos by meaningful event or situation 
groups [19]-[24], Especially in [19] and [20], taken date/time 
stamp feature and color features have been combined together 
to cluster photos by events in an automatic manner. In general, 
user demand for event indexing tends to exhibit little coherence 
in terms of low-level visual features, though camera metadata 
could help to organize photos in more semantically meaningful 
event groups. We also developed, in our previous studies [23], 
[24], a situation-based photo clustering approach by combining 
taken-date/time stamp of camera metadata and low-level visual 
features, where situation stands for similar background scenery 
taken in a close proximity of time. 

Especially for semantic photo categorization, Boutell et al. 
in [25], have proposed a probabilistic approach to incorporate 
camera metadata with low-level visual features. They exploited 
a useful set of camera metadata, which is related to flash, scene 
brightness, subject distance and focal length, and verified it in 
indoor/outdoor, sunset/non-sunset, and man-made/natural scene 
classification. Boutell's method has one major disadvantage on 
the applications to generic photo classification. As presented in 
the study, it has limited application to a few global scenes since 
it used only global features. A photo usually contains many local 
semantics that are located in a local region. So, to extend the use 
of camera metadata to the classification of many other local and 



global photo semantics, camera metadata is strongly required 
to be incorporated with visual features of local photo region. 
For example, let us see a photo containing a human face in fore- 
ground behind background scenery. If its camera focus is on the 
person, then subject distance and focal length would be probably 
short. Given this condition, Boutell's classifier may have a diffi- 
culty of detecting background scenery in spite of the additional 
use of low-levet visual features for the classification. 

Many literatures have dealt with multilayered approaches for 
multilabel classification. In particular, multilayered SVM has 
been also adopted for multilabel image classification. However, 
multilabel detection problem with camera metadata is missing, 
up to the present. Thus, one remaining challenge for multilabel 
detection in photo is to model classifiers in the multilayered 
scheme. Here, the classifiers should be to incorporate regional 
low-level features with camera metadata so that they can capture 
multiple local photo semantics. From this idea, in this paper, we 
exploit a novel multilayered classification model that combines 
camera metadata, low-level features, and intermediate level of 
semantic features, especially for multilabel classification. The 
proposed method is composed of two-layered SVM classifiers. 
The two-layered SVM classifiers operate to discover local and 
global photo semantics in a feed-forward way. The first layer 
aims at measuring the likelihood rate of predefined local photo 
semantics based on both camera metadata and local low-level 
visual features. Local photo semantics provide an intermediate 
level of photo semantics as bridging the semantic gap between 
low-level features and global, i.e., high-level, photo semantics. 
In the second layer, we determine one or more potential global 
photo semantics based on the likelihood ratio of local semantics. 
To construct a SVM classifier producing a posterior probability, 
a parametric model is applied to fit the output confidence of the 
SVM classifier to posterior probability. We also exploit concept 
merging based on a set of semantic-confidence maps in order 
to cope with selecting more likelihood local photo semantics on 
overlapping photo regions. 

II. Schematic Model of Home Photo Categorization 
A. Overview 

Fig. 1 illustrates the schematic model of the proposed home 
photo categorization, given by two separate approaches. One 
approach is to incorporate camera metadata, such as aperture 
number, exposure time, flash-fired, etc., with low-level features, 
such as color, texture, shape, and so forth. Camera metadata is 
not obtained from the photo content itself; rather from picturing 
devices such as cameras and camcorders. The other approach is 
to stepwise model photo semantics in a way of combining local 
and global semantics, and to use the models to classify photos 
into multiple semantic concepts. For this, we predefine a set of 
local and global semantic concepts. The local semantic concept 
stands for lower level photo semantics while the global concept 
stands for higher level of photo semantics. A global concept can 
be linked with one or more local concepts. The strength of the 
linkage is represented a probability. In general, the use of lots of 
probability models in multilayered structure often causes error 
propagation due to the multiplicative of probabilities. But, the 
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Fig. 1. Proposed two-layered categorization scheme. 



use of different feature space for each layer alleviates the error 
propagation. 

Camera metadata is considered at the level of local semantic 
classification so that it copes with regional low-level features. In 
general home photo, visual semantics are exhibited on different 
spatial regions. If camera metadata is combined with low-level 
features extracted from a holistic photo region, they might be 
strongly associated with only dominant low-level features of the 
photo, thus facilitating the detection of a single major semantic. 
This could be a problem on multilabel categorization. In order 
to solve the aforementioned problem, we embody the proposed 
home photo categorization method as a two-layered approach 
composed of local and global semantic classification processes. 

Fig. 2 shows overall procedure of the proposed home photo 
categorization scheme. The proposed method consists of eight 
steps. 1) If a photo to be classified is input. 2) then its camera 
metadata is extracted from the photo file. 3) Entire region of the 
photo is spatially divided into several local regions. 4) Multiple 
low-level visual features are extracted from the local regions. 5) 
Each local region is represented by a combined feature vector 
constructed with its camera metadata and low-level features. 6) 
Then, the local features are fed into local semantic classification 
to detect local photo semantics by measuring joint probability 
of the local regions features about the models of local semantic 
concepts. 7) A local semantic feature vector is generated by a 
concept merging process that makes the local regions keep the 
highest probable concepts. 8) The local semantic feature vector 
is fed into global semantic classification to detect global photo 
semantics by measuring joint probability of the local semantic 
features about the models of global semantic concepts. 

B. Camera Metadata for Home Photo Classification 

Camera metadata is usually provided from the Exif header of 
photo. Most current digital cameras support Exif specification 
for the camera metadata. It is only available for JPEG photos. 
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Fig. 2. Overall procedure of the proposed home photo categorization. 



But, this is not matter since the JPEG is the most common photo 
format supported in all consumer cameras. The Exif includes 
hundreds of metadata tags. Some of the Exif information, such 
as exposure time, shutter speed, aperture number, flash setting, 
subject distance, and focal length, are discriminative in specific 
classes [20]. In sense of usefulness, the camera metadata can be 
grouped into three families: scene brightness, subject distance, 
and flash setting (absence or presence). The three classes can be 



YANG eta!.: SEMANTIC HOME PHOTO CATEGORIZATION 



considered mutually independent each other. Scene brightness 
includes three tags: exposure time, aperture number, and shutter 
speed. In general, outdoor sunlighting is brighter than indoor ar- 
tificial lighting. The exposure-time is closely correlated with the 
shutter speed. Both exposure-time and shutter speed would be- 
come shorter at a brighter scene while longer at a darker scene. 
The subject distance can be estimated by a camera in auto-focus 
mode, which becomes usually longer at landscape scenes while 
shorter at object-oriented scenes. However, the subject distance 
is dependent of the position of main subjects, so it seems rather 
useless for general scene classification. For example, the subject 
distance is short when camera focus is on a certain foreground 
subject with landscape in the background. Intuitively, the flash 
setting seems useful in that it would be much more often used 
in case of lack of lighting such as indoor and night scenes. 

Our major concern about the use of camera metadata in photo 
categorization is a way to combine with other features, such as 
low-level features. The camera metadata usually presents global 
scene characteristics although some high-end cameras allow to 
preserving light condition restricted on some parts of the whole 
picture. In multilabel detection, many image semantics could be 
exhibited over small subrcgions of photo. This means that the 
main subjects of photographer's interest might be on a small 
subregion. The low-level features of subregion are definitely 
useful to derive high-level photo semantics. Accordingly, the 
camera metadata should be considered to be combined with the 
regional low-level features to better derive the high-level photo 
semantics. The proposed two-layered scheme enables to realize 
this combination as well as to preserve advantages of the use of 
layered photo semantic features, i.e., local and global semantic 
features, in multilabel photo classification. 



III. Local Semantic Classification 



A. Regional Division for Local Semantics 

Most current digital cameras support auto-focusing (AF) 
mechanism that works as moving the camera lens in and out 
until the sharpest possible image of the subjects is projected 
onto the image receptor. AF system provides a certain number 
of censoring regions, of which some regions of interest can be 
selected by photographer. A censoring region typically forms a 
rectangle. It means that photographer's intension in capturing 
photo may be found in the rectangular censoring regions. 

The best representation of local image semantics is given by 
a semantic object segmentation, which produces elaborate ob- 
ject contours or details. In [30], object regions have been found 
in a hierarchical way based on a partition tree. But, up to the 
present, there seems no almighty method for object segmenta- 
tion. Most related approaches are quite expensive in computa- 
tion and even sometimes produce incomplete result in complex 
natural images. 

So, instead, we propose a simple block segmentation to cap- 
ture visual semantics that appear on local photo regions. The 
block segmentation is relatively inexpensive. However, to over- 
come low segmentation performance, we design a finite set of 
region templates, in sense of the rectangle censoring system of 
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Fig. 3. Photographic region templates. 



digital camera. Fig. 3 shows proposed photographic region tem- 
plate, denoted as PRT. Although the PRT is built by block tessel- 
lation with a fixed number of blocks, we presume that it could be 
fast and good enough to capture photographer's intension. The 
fundamental observation behind the PRT is that mainly-con- 
cerned subjects are typically focused, taking a larger portion 
and being sharper than other unconcerned subjects. Thus, many 
other most likely small, blurred subjects are often out of concern 
in the photo. In order to build meaningful region templates, three 
conditions are considered: 1) the PRT should be large enough to 
detect visual semantics in the local photo region; 2) it should be 
small enough not to be time-consuming in feature extraction; 
and 3) it should support spatial scalability to detect semantic 
subjects in a wide range of spatial scale. 

The region template is composed of ten local photo regions: 
one center region (#! in Fig. 3), four corner regions {R2, -R3. 
i? 4 , and i?5 in Fig. 3), two horizontal regions (R& and R 7 in 
Fig. 3), two vertical regions (Rq and Rq in Fig. 3), and a whole 
photo region (R w in Fig. 3). The four corner regions are parts 
of the vertical, horizontal, and the whole photo regions. And 
the center region overlaps partially with the corner, vertical and 
horizontal regions, and entirely with the whole photo region. 
In this paper, non-overlapping region elements including one 
center and four corner regions are referred to as "basis regions." 
Fig. 4 shows some examples with different photographic com- 
position, which are localized by the region templates. In the first 
example on the left side, the photo could be well-represented 
using three local regions: the top-vertical region showing sky, 
left-bottom corner region showing lake, and right-bottom corner 
region showing field, where the sky, lake, and field can be con- 
sidered as local semantic concepts of the first example photo. 

B. Integration of Camera Metadata and Local Visual Features 

To integrate camera metadata with local visual fea- 
tures in the proposed photo categorization, we first gener- 
alize the following probabilistic feature combination. Let 
X = {xx 7 x 2 , ..-,xn} be a finite set of N local semantic 
concepts assumed to frequently appear in home photos. And, 
let F cam = {/cami /cam; ■ • • t /4m} De a finite set of ^ camera 
metadata and F}£ al = {/^, . . . , /^ v } be a finite set of 
J low-level features obtained from local photo regions. Then, 
the likelihood rate of a semantic class of x n € X given feature 
vector F looal = {F cam . Fj°° a1 } can be denoted as the joint con- 
ditional probability like P(x n \F 1 ™ 1 ) = P(x n \F am ,'B\£ 1 ). 
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Fig. 4. Examples of local regions segmented by the proposed region template. 



By the Bayesian theorem, the joint conditional probability can 
be decomposed as follows: 

P(x n |F local ) = P( a ; n |F cam) F^ al ) 

P(Sn)^(F cam ,F}r L K) 

- P(F cam ,F[^) • 

Here, the camera metadata is independent of the regional low- 
level visual features. So (1) can be written again as follows: 

P{XnlF } ~ P(F cam )-P(F^) • (2) 

This probability derivation presents the basic methodology for 
integration of camera metadata and regional low-level visual 
features. The integrated feature is used only in the first layer, 
so that we can predict likelihood in relation of a predefined set 
of local photo semantics. A local semantic feature is presented 
by degree of strength of the local photo semantics in the first 
layer. 

A comparative probability of P(a; n |F Iocal ) can be 
P(x n \F&° hal ), which is denoted as P(x n \F c * m , Ff^ bal ) 
where Fj£° baI is a finite set of low-level features obtained from 
a holistic photo region. As studied in the Introduction section, 
the representation and understanding of local semantics is 
necessary for multilabel detection. For ~ix n ,x n e X, since x n 
stands for a local semantic concept implying low-level photo 
semantics, F 10011 would have better representation of local 
photo semantics than would Fe lobal . Accordingly, we expect 
that P(a; Tl |F local ) > P(x n |F slobal ) whenever x n belongs to 
the true category that is located on the subregion of the local 
low-level features. Although F sloba! can be directly employed 
for global semantic classification, it is not applicable in the 
two-layered approach. 

C. Local Semantic Learning 

SVM is employed as local semantic classifier in the first layer. 
It gives a good binary classifier that is used to find the decision 
function of optimal linear hyper-plane given labeled training 
data. The hyper-plane is linearly separable in the feature space 
(h). The input feature in the feature space (F) is mapped onto 
the feature space via a nonlinear mapping (tp : F -+ h), al- 
lowing one to perform nonlinear analysis of the input data using 
a linear method. In general SVM, a kernel is designed to map 
the input / space to the feature space. With the "kernel trick" 
property [11], [12], the kernel can be regarded as a similarity 

Hi 



measure between the two feature vectors without explicit com- 
putation of the map tp. Using the kernel function, the SVM 
classifier can be trained with training feature data. For this, an 
optimal hyper-plane is found to correctly classify the training 
data. By the optimization theorem of SVM, the decision func- 
tion (#Jf cal ) can be formed to predict the local concept (ar„) of 
the input local feature vector (F !ocal ) as follows: 



^locai^iocaij = £{w n (t) • z n (t) • K (F n (i) , F local )} + a n 

(3) 

where K is kernel function that can be a linear, polynomial, or 
radial-basis function. And, F n (t) is the tth support vector of the 
hyper-plane for x n , w„ is the corresponding weighting vector 
of the support vector, z„ is the corresponding class vector of the 
support vector, and a„ is the threshold vector optimized for x n . 

In order to construct SVM classifier to produce a posterior 
probability, the output confidence of the SVM classifier is fitted 
to a parametric sigmoid model. The parametric sigmoid fitting 
model for the classifier of a local photo semantic *„ forms as 
follows: 



P {x n = i|$^(F ,oca1 )) 



f exp (A n ■ $£ cal (F^ al ) + B n ) 



(4) 



where A n and B n are parameters to determine the shape of 
the sigmoid model for the local concept (x% cal ). So, the SVM 
output ranged from -00 to 00 can be fitted to the probabilistic 
output ranged from 0 to 1 . 

The best parameter set of (A, B) is measured by solving 
the following regularized maximum likelihood problem 
with a set of labeled training examples. Given a training 
set (#}° caI (Fj),a:i), let us define a new training set 
($Jf cal (Fi),a:.), where the x[ is a target probability value. 
The new target value is used instead of (-1, 1) for all of the 
training data in the sigmoid fit. This aims at making the new 
target value converge to (0, 1) when the training set size ap- 
proaches infinity. The new target value x\ is defined as follows: 



•i = 1 



where 0+ is the number of positive samples and 0_ is the 
number of negative samples. Then, the best parameter set for 
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a local photo semantic is obtained by n 
cross-entropy error function 



g the following local photo regions. If v R is replaced by P(x n \F R ), (1 1) can be 
written again as follows: 



argrain -V {*J -lagft + (1 -sj) ■ log(l < 6 > V={«i,...,f 
(A,B) [ i J 

where pi denotes P(xi|$^ cal (Fi)). We adopt an optimization 
method [17] to find the optimized parameters minimizing the 
above error function. 

D. Local Semantic Classification 

The first step for local semantic classification is to divide the 
whole region of an input photo into ten subregions by the PRT. 
Multiple low-level visual features are extracted from each local 
region and fed into the local concept detectors. For the local 
concept detection, let R = R 2 , . . - , Rio} be a set of the 
local regions. Then, the features of the local regions (R) are 
denoted as F^ = {Ff am ,Fg w }. (2) and (3) can be specified 
for the local regions as follows: 



= UIR 



P(z n |F*)=P(a„|F cami F£ w ) 

_ P(x n ).P(F cs , m \x n ).P(F R Jx n ) 
P(F cam ) • P (F£ w ) 

where F cam is the same over all local regions in an input 
photo and P(F£ w |a;„) is a likelihood rate of F£ w about x n . 
P(F^ w |a; n ) is estimated by a sigmoid model as follows: 



P (* n |F£ w ) = 



1 + exp (A n • $1?^ (F£J + B n ) ' 



IV. Global Semantic Classification 

A. Regional Semantic Merging 

To find more likelihood semantics on the overlapping local 
regions, a concept merging is performed. It operates based on 
a semantic confidence map (SCM). The SCM is used to keep 
the most confident concept for basis local region set, denoted 
as R basis . The R ba5is consist of one center and four corner re- 
gions, i.e., {Ri,R2,R3,Ri,R&h where ri S htl y Rbasis c R " 
The SCM is a set of five groups of overlapping local regions 
as shown in Fig. 5. SCM 1, denoted as Rf ap , includes Ri and 
Rio regions. SCM 2, denoted as R™ 1 *, includes R 2 , Re, Rg, 
and .Rio regions. SCM 3, denoted as R™ ap , includes i? 3 . R&, 
Rs, and R w regions. SCM 4, denoted as R™ ap , includes /? 4 , 
J? 7> Rs, and R i0 regions. SCM 5, denoted as R l ™ p , includes 
Rs,R 7 , Ro, and Rn> regions. Then, the confidence value of a 
local concept (x n ) of the a basis region (R b e R basls ) is calcu- 
lated as follows: 

^ = ma X (^| J ReRr P ) d3) 

where, v\ = max(v*\R € {R 2 , Re, Rs,Rw}) in case of R 2 . 

Given a basis region R b (e R basis ), (13) can be written again 
as follows: 



Likewise, P(F cam |z„) is a likelihood rate of F cam about x n . 
So, it is also estimated by a sigmoid model as follows: 



POnlFcam) S 



1 + exp (A n ■ #J«» 1 (P« a ua) 4- B n ) ' 



The probability set of x n over all local regions (R), denoted 
as P„, can be written as follows: 

P„ = {P{x n \F R i),P{x n \F R >), . . . ,P{x n \F R »>)} 

10 

= {JP{x n \F Ri ). (10) 

The probability set about the local concept set (X), denoted 
as Px, can be written again as follows: 

P X = \JPn= [J l)P(x n \F^). (11) 

Then, given a local region, a local semantic vector with the 
probability values is composed for each local concept. The local 
semantic vector presents the likelihood of local semantics on 



B. Association of Local Semantics With Global Semantics 

Global concept is a relatively higher level of semantics than 
all local concepts. Fig. 6 shows an example of the relationship 
between local and global photo semantics. There are eight local 
concepts (sky, tree-wood, flower, rock, bridge, windows, streets 
and building) and two global concepts (terrain and architecture). 
In sense of visual semantics, terrain is strongly linked with sky. 
tree-wood, flower, and rock while it is loosely linked with street 
bridge, windows, and buildings. On the contrary, architecture is 
strongly linked with bridge, windows, street, and building while 
it is loosely linked with sky, tree-wood, flower and rock. 

From this observation, we consider the degree of strength of 
the semantic link between local and global semantic concepts. 
The higher value presents a stronger connection between local 
and global concepts. This approach could bridge the semantic 
gap between low-level feature and high-level semantic concept. 
Let Y = {yi, y 2 , Vm) be a set of M global concepts, sub- 
ject to M < N. Global concepts are trained based on the like- 
lihood of local SVM classifiers. The decision function (#„,) 
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LOCAL SEMANTICS 




Fig. 6. Example showing the relationship between local semantics and global 
semantics. 



to predict a global concept (y m ) of input local semantic vector 
(V 6 ) given a basis local region (ify) is formed as follows: 

a*b*(v*) = Y,v m (sK m ( S )-K {V m (s),V b )+a m (15) 

where V m (s) is sth support vector of the hyper-plane for y m , 
Wm is the corresponding weighting vector of the support vector, 
£ m is the corresponding class vector of the support vector, and 
a m is the threshold vector optimized for y m . 

The V fc constructs a semantic feature and it is fed into the 
global concept classifier. Like local semantic classifier, the joint 
conditional probability of a global concept y m € Y given the 
semantic feature V b vector for a photo region can be estimated 
by sigmoid fitting as follows: 

P(y m \V b )=P{y m \vlv b 2 ,...,v b N ) 

7 TT1 V < 16) 

l + exp(A m .*5S! ob " I (V») + B m ) 

M2> 



where A m and B m are parameters to determine the shape of the 
sigmoid model for the global concept (y m ). 

C. Global Semantic Classification 

Given a basis local region set, the merged confidence values 
for all local concepts are used to classify the local regions into 
the target classes. Since one of the main objectives is to detect 
multilabel detection meaning that an input photo can be labeled 
by one or more classes, we propose a criterion for multilabel 
categorization. Potential categories of the photo are determined 
given the probability values for the five basis local regions of an 
input photo. First of all, the probability values for all basis local 
regions are aligned in ascending order. 

Then, the top-if classes with respect to probability value are 
assigned to classes of the input photo, whose probability values 
should be close enough, i.e., higher than a threshold. The clas- 
sifier assigns the class of an input photo to multiple global con- 
cepts that should be satisfying the following condition, given by 

c m = m , itP(y m )-P(y m \V b )>Pa, 

for any class and any basis region (17) 

where c m is the predicted class of the input photo and P t h is the 
heuristic threshold value for the categorization. 

V. EXPERIMENTS 
Experiments were performed with 3086 photos that comprise 
the official test database of the MPEG-7 visual core experiment 
2 (VCE-2). The VCE-2 aims to achieve photo categorization by 
using MPEG-7 visual descriptors. It also offers corresponding 
ground truth (GT) set, which has been cross- verified by several 
VCE-2 participants who had taken them. The GT set is given by 
seven semantic categories popularly appeared in home photos: 
"architecture," "indoor," "waterside," "nightscene," "snows- 
cape," "terrain" and "sunset." In order to cover any preference 
of human visual perception, the GT set has been strictly made 
in detail. That is, an important rule in the GT decision was that 
a photo could be assigned with any semantic concept of which 
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Category 


Pi a 


P2 b 


L-ratio (=p|/p2) 


Architecture 


1280 


2370 


1.85 


Indoor 


803 


1047 


1.30 


Terrain 


1297 


2499 


1.93 


Night 


372 


758 


2.04 


Snowscape 


175 


336 


1.92 


Sunset 


70 


193 


2.76 


Waterside 


650 


1426 


2.19 



"pi is the number of true photos that belong to the category defined in the 
left-most column; *pj is the number of true categories of the photos denned 
in the second column. 

a scene could be found even in very small portion of photo. 
Thus, any of the test photos was often labeled by multiple true 
classes. It should be noted that the strictly-made GT set may 
cause some degradation in categorization performance. 

Table I shows the statistics of the test photos. The test photos 
mostly belong to architecture or terrain category since they were 
usually taken during ordinary events. A photo exhibits about two 
categories on average and about four on maximum. We mea- 
sured how many other categories appear with a certain category 
at the same time. To numerically quantify this relation, we ex- 
ploited a criterion, called .L-ratio, where L stands for local. For a 
category, the L-ratio is calculated by a relative rate of the number 
of true photos belonging to a certain category over the number 
of true categories of the photos. So, the bigger .L-ratio of a cat- 
egory is, the more categories appear in the test photos at the 
same time. As can be seen in Table I, indoor and sunset showed 
the lowest and highest L-ratio, respectively. This means that in- 
door would often appear alone on the test photos while sunset 
would often appear with other categories. For example, as seen 
in Tabic I, there are 1280 photos that belong to the architecture. 
The 1280 photos contain 2370 true categories including archi- 
tecture and any others. 

In the experiment, 1597 photos were used for training. They 
were also from the MPEG-7 VCE-2 official training data, which 
is totally independent of the test data set. Of the training photos, 
800 photos were from general home photos and 797 photos 
were from the Corel photo collection. To construct a set of 
local concept, we built nine important families of concepts that 
would frequently appear in local regions of home photos. The 
families of the local concepts consist of "ground," "human," 
"indoor," "mountain," "night," "plants," "sky," "structure," and 
"water." The concept families are subcategorized to the 34 
local concepts as follows: "ground" to "gravel," "park," "pave- 
ment," "road," "rock," "sand," and "sidewalk;" "human" to 
"face" and "people;" "indoor" to "indoor" and "indoor-light;" 
"mountain" to "field," "peak," and "wood;" "night" to "night" 
and "street-light;" "plants" to "flowers" "leaves," and "trees;" 
"sky" to "cloudy," "sunny," "sunset," and "sunset-on-moun- 
tain;" "structure" to "brick," "arch," "building," "wall," and 
"windows;" "water" to "high-wave," "low-wave," "still water," 
"mirrored water," "beach," and "ice (snow)." 

For training local concepts, we patched the training photo to 
local regions and then selected positive and negative samples by 
human visual perception. Then, multilow-level visual features 
were extracted from the patched photo database. Several color 
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and texture descriptors of the MPEG-7 were considered for the 
feature extraction [26], [27]: color structure (CS), color layout 
(CL), and scalable color (SC) descriptors are used for color fea- 
tures; and homogeneous texture (HT) and edge histogram (EH) 
descriptors are used for texture features. 

The sigmoid model parameters were calculated for each local 
semantic classifier. Fig. 7(a) shows the histogram of positive 
and negative samples for indoor classifier. The solid line is the 
probability of negative samples while the dashed line is that of 
positive samples. As shown in Fig. 7(a), the histogram is not 
Gaussian, probably due to the small number of training data. 
Fig. 7(b) is derived by using Bayes' rule on the histogram es- 
timates of the class-conditional densities. As can be seen, the 
sigmoid fit works well with relatively small number of training 
samples. 

We measured performance with accuracy and preci- 
sion since the numbers of the test data are imbalanced 
over all classes. As in general definition, accuracy = 
(TP + TN)/ (total number of samples), and precision = 
TP/ (TP + FP), where TP, TN, FP, and FN stand for "true 
positive" when the case is positive and predicted positive, "true 
negative" when the case is negative and predicted negative, 
"false positive" when the case is negative but predicted positive, 
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Fig. 8. Local semantic classifier for the camera metadata, (a) Indoor-outdoor 
classifier, (b) Night-daytime classifier, (c) Combination of the two classifiers to 
detect local photo semantics. 



and "false negative" when the case is positive but predicted 
negative, respectively. 

For the camera metadata, we selected exposure-time (ET), 
aperture number (AN), local length (FL), and flash-fired or not 
(FF). As seen in the Boutell's studies, useful camera metadata is 
related to scene brightness. In our experiment, we also see that 
the camera metadata is useful only in a limited set of categories 
classified by scene brightness. However, many local semantics 
are not directly related to scene brightness. In order to extend 
the use of the camera metadata to all potential local semantic 
classifiers, we divide the local classifiers into night/daytime and 
indoor/outdoor classifier groups. From this way, the estimated 
posterior probability of the camera metadata can be assigned for 
all local classifiers. Fig. 8 shows the local semantic classifiers 
for camera metadata. Fig. 8(a) shows indoor/outdoor classifier 
that outputs probability values about indoor and outdoor classes. 
Likewise, Fig. 8(b) shows night/daytime classifier that outputs 
probability values about night and daytime classes. To associate 
the two camera metadata classifiers with the 34 local concepts, 
we make a classification scheme as seen in Fig. 8(c). As such, 
the first step is to classify the input camera metadata into in- 
door or outdoor classes. The indoor probability is assigned to 
indoor classes while the outdoor probability is assigned to out- 
door classes. The second step is to classify the input camera 
metadata into night and daytime classes. The night probability 
is assigned to all night classes, and the daytime probability is 
assigned to all daytime classes including ground, human, moun- 
tain, plant, sky, structure, and water classes. 



Combination of 
camera metadata 



Average performance a 



sategorization (%) 



ET + FF 
ET + AN 
ET + AN + FF 



73.07 
74.63 
26.09 
74,40 
74.60 
81.89 
82.57 



Night/daytime 
categorization (%) 



"Average performance is measured by averaging recall and accuracy. 

For the purpose of choosing the best combination of camera 
metadata, categorizations with different combination of camera 
metadata were performed in indoor/outdoor and night/daytime 
classes. Before measuring the categorization performance, we 
also measured Kullback-Leibler (KL) divergence [28], [29], as 
studied in [25]. In our result, the combination of the ET, AN, 
and FF also showed the greatest KL divergence, whose values 
were about 2.30 and 1 .96 in indoor/outdoor and night/daytime 
classification, respectively. 

Table II shows categorization performance with only camera 
metadata. As expected from the KL-divergence measured 
above, ET, AN, and FF gives the best combination of camera 
metadata in both indoor/outdoor and night/daytime classes. 

To verify the proposed combination of camera metadata and 
local visual semantics, experiments were performed with three 
different cases: 1 ) categorization by using only global low-level 
features; 2) categorization by using a combination of regional 
low-level features and local semantic features on the proposed 
two-layered scheme; and 3) categorization by the additional use 
of syntactic camera metadata on top of the second case. Table III 
shows a comparison of categorization performances by the three 
cases. As predicted in our study, overall performance is low in 
the case 1). Although some global categories, such as night, 
sunset, and snowscape showed relatively higher performance, 
other local categories did not. That is, global semantics would 
be captured better than local semantics. Using a combination 
of regional low-level features and local semantic features leads 
to more than 5% increase of the performance. In this case, the 
proposed scheme was adopted, but there was no use of camera 
metadata. This means that the proposed local semantic features 
provide more useful cues for local photo semantic classification. 
Finally, when we used syntactic camera metadata on top of the 
second case, the performance was more enhanced. There was 
about 5% increase in architecture category, about 7% increase 
in indoor category, about 5% increase in terrain category, about 
2% increase in night category, about 1% increase in snowscape 
category, about 2% increase in sunset, and about 2% increase in 
waterside category. Fig. 9 shows categorization performance of 
the proposed method over various thresholds. 

The proposed method was compared to a major referenced 
work using Bayesian network classifier based on global visual 
features and camera metadata [25]. The main difference of the 
proposed method from the Boutell's one is that we provide a 
multilayered scheme to model local and global semantics for 
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TABLE III 

Performance Comparison of Categorization Schemes With Different 
Features 





Recall / accuracy (%) 




Category 


(a) With only 


(b) With only 


(c) With local 




global low-level 
features 


local semantic 
features 


semantic features & 
camera metadata 


Architecture 


61.01/61.08 


66.43/67.82 - 


70.13/72.23 


Indoor 


63.98/64.21 


75.34 / 74.93 


83.59/82.33 




67.52/67.60 


74.42/72.73 


80.24/77.48 


Night 


89.53/89.71 


91.57/89.85 


94.38/92.32 


Snowscape 


75.76/74.84 


82.23/83.70 


83.10 / 84.56 




76.47/77.41 


78.77/81.29 


81.67/ 82.54 


Waterside 


67.64 / 68.02 


72.16/71.34 


74.32/72.72 




0 0.1 0.2 0.3 0.4 0.5 0.6 0.1 0.8 0.9 1 

Recall rate 



Fig. 9. Recall-accuracy curve of the categorization with local semantic and 
camera metadata. 

multilabel detection. Boutell's method, in [40], was concerned 
in that it would be weak in multilabel classification. Thus, our 
presumption is that the proposed method would outperform the 
conventional one, especially Boutell's method, in multilabel 
classification if the advantages of the proposed two-layered 
S VM classifiers have been realized as it should. Table IV shows 
categorization performance of the two different methods. The 
training and testing data was the same as the above experiment. 
As seen in the result, almost categories except the architecture 
category were better detected by the proposed method than by 
the conventional method. In indoor and terrain, both methods 
showed similar performance. But, the proposed method much 
better detected other categories such as night, snowscape, 
sunset and waterside. As shown above, architecture, terrain, 
and indoor categories showed low i-ratio, i.e., the categories 
would often appear alone. On the contrary, night, snowscape, 
waterside, and sunset categories showed relatively high L-ratio, 
i.e., they would often appear with other categories. It means 
that the proposed method would be useful to detect multiple 
categories of generic home photos, thus providing more general 
applications to photo categorization. 

For multilabel detection, we compared categorization error 
rates according to the number of true categories per photo. 
Fig. 10 shows the categorization error rates in the proposed 
method using two-layered SVM, where the number of true 
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TABLE IV 

Performance Comparison With Bayesian Network Scheme 



Category 


Bayesian network 




Proposed two-layered SVM 


Recall 


Accuracy Average 


Recall 


Accuracy Average 


Architecture 


87.34 


70.97 


79.16 


70.13 


72.23 71.18 


Indoor 


96.64 


71.93 


84.29 


83.59 


82.33 82.96 


Terrain 


90.28 


66.32 


78.30 


80.24 


77.48 78.86 


Night 


84.14 


67.79 


75.97 


94.38 


92.32 93.35 


Snowscape 


87.43 


41.95 


64.69 


83.10 


84.56 83.83 


Sunset 


84.29 


58.59 


71.44 


81.67 


82.54 82.11 


Waterside 


73.08 


55.41 


64.25 


74.32 


72.72 73.52 




0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.3 1 

Threshold 



Fig. 1 0. Categorization error rate with respect to the number of true categories 
appeared in a photo. 



categories per photo is ranged from 1 to 4. The error rate was 
calculated by (1.0-accuracy). Since the outputs of the SVM 
classifiers were fitted to the associated sigmoid model, the 
threshold is ranged from 0.0 to 1.0. In most cases, the minima 
of the error rates can be found around 0.5. This means that the 
SVM classifier has been well-trained to generate a probabilistic 
output. As seen in Table V, we compared the proposed method 
to Boutell's one in the categorization error rate. Since Boutell's 
method uses the maximum a posteriori (MAP) criterion, it 
probably gives only one categorization result. In the case of the 
proposed method, the error rate was about 1 1 % when photos 
exhibit only single visual category. Even though the error 
rate increased when the number of true categories per photo 
increased, the amount of the increase of the error rate gradually 
decreased, e.g., the error rate increased by about 3% when the 
number of true categories increased to 4 from 3. In the case 
of Bayesian network, the error rate increased over all numbers 
of true categories per photo, compared to that in the case of 
the proposed method. When the number of true categories per 
photo increased, the error rate of the Bayesian network method 
was more degraded compared to that of the proposed method. 
This is because Boutell's method combined metadata with 
global low-level features. We see that the proposed method 
would outperform the Boutell's method in the multiclass clas- 
sification. 

Finally, we also performed comparative experiments with a 
k-NN approach [41]. The k-NN approach was performed for the 



IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 17, NO. 3, MARCH 2007 



TABLE V 

Comparison of Categorization Error Rates According to the Number 
of True Categories Appeared in a Photo 



Number of true categories / photo 


1 


2 


3 


4 


Minimum error rate (%) for the 
proposed two-layered SVM 
scheme (corresponding threshold 
at the minimum error rate) 


11.30 
(0.66) 


23.40 
(0,53) 


28.44 
(0.49) 


31.69 
(0.46) 


Minimum error rate (%) for 
Bayesian network scheme 


18.11 


30.44 


43.69 


54.72 



TABLE VI 

Performance Comparison With k-Nearest (k-NN) Scheme 



Category 


Performance of k-NN 
method (%) 


Architecture 


60 


Indoor 


52 


Terrain 


65 


Night 


65 


Snowscape 


51 


Sunset 


25 


Waterside 


35 



MPEG-7 VCE-2, It also used five MPEG-7 visual descriptors 
(SC, CL, CS, HT, and EH), which is the same as our method. 
The difference of this method from ours is that it modeled k-NN 
classifiers with only global low-level visual features. The k-NN 
approach did not utilize any camera metadata and multilayered 
classification with local and global semantics. Table VI shows 
performance comparison of different learning and classification 
schemes. As shown in the result, the average performance of the 
k-NN method was much lower than that of the proposed method 
over all categories. This simply shows that SVM based on SRM 
outperforms than k-NN based on ERM in generic classification. 



VI. Discussions and Conclusions 
We exploit a photo categorization scheme that utilizes both 
camera metadata and semantic features in two-layered scheme. 
The camera metadata would provide useful cues independent of 
photo contents as combined with low-level visual features for 
the purpose of measuring the likelihood rate about local photo 
semantics. However, error accumulation is problematic when 
using many probability models. The error accumulation often 
results from feature diversity and high dimensionality, and thus 
it is expected in the proposed two-layered feed-forward SVM 
classifiers. However, the proposed two-layered scheme reduces 
the error accumulation since it performs each classification in 
totally different two feature space; one is low-level feature space 
and the other is local semantic probability space. The error ex- 
hibited in the first SVM layer can be also reduced in the second 
SVM layer since the second SVM layer considers only corre- 
lation between local photo semantics, which is estimated from 
low-level feature, rather than the low-level feature itself. In ad- 
dition, it is possible that there is no metadata in some photos. 
However, it is not very common in newly-acquired photos from 
users. Our target application is a home photo album that en- 
ables semantic photo categorization without users' manual ef- 
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forts. In this situation, users may not intentionally remove the 
metadata because they probably want to enjoy that kind of en- 
hanced album functionality in organizing their photos for con- 
venience. 

The efficacy of the proposed classification method was 
demonstrated with 3086 home photos from MPEG-7 VCE-2 
official databases. Compared to a major conventional method 
using global low-level features and camera metadata modeled 
by a Bayesian network, the proposed two-layered SVM per- 
formed better to classify visual semantics for general home 
photos. However, we need to do in-depth and detailed analysis 
on the profits due to the multilayered approach with camera 
metadata for multilabel detection in future work. Furthermore, 
we need to employ other syntactic features, such as keyword, 
spatial context, user preference, etc., for the proposed photo 
categorization scheme, and optimize the features to enhance 
the current performance. 
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Modulation and Coding for 
Information Storage 

The success of certain modulation and coding techniques in data 
communications have inspired promising new applications in 
digital data storage. 
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is industry is con- 
| cerned with reliable and efficient 
is for transmitting information 
I from one place to another. The infor- 
mation storage industry is concerned 
I with reliable and efficient means for 
transporting information from one time to anoth- 
er. The teachings of information and communi- 
cations theory apply equally well to both scenarios. 
In communications as well asstorage systems, infor- 
mation must be transported through noisy channels 
that accept input signals (perhaps from some 
constrained class of signals) and produce output sig- 
nals from which the transmitted or stored infor- 
mation must be recovered. Yet the readers of 
this magazine are, for the most part, unaware of 
the applications of (heir skills to the storage 
industry, wh ich has worldwide annual sales in excess 
of $50 billion. {This figure is for magnetic stor- 
age alone.) 

While communicators are concerned with 
maximizing the rate (in bits persecond) whereby dig- 
ital information can be transmitted and reliably 
received, storage researchers are largely con- 
cerned with maximizing the arcal density (in bits per 
square inch) and volumetric density (in bits per cubic 
inch)forstoringandreliablyretrievinginformation. 

The storage industry has made steady progress 
increasing the density of information storage of 
digital data. Over the last 25 years, the areal den- 
sity of Storage of digital data on magnetic hard 
disks has grown geometrically atacompoundgrowth 
rate of abou 1 29%. This remarkable growth is reflect- 
ed in Fig. 1, which shows the areal density of 
high-end IBM disk drives as a function of produc- 
tion year. 

Most of this increased storage density has 
resulted from improvements in the part of the 
system that we call "the channel," including the stor- 
age medium itself, the read and write heads, mechan- 
ical features determining the flying heights and 
positioning of these heads, and so on. 

Though less dominant, the contribution of 
advances in signal processing and coding has not 
been insubstantial. For example, channel model- 
ing indicates that without the improvements in 



run length-limited codes (described in more 
de tail later) the compoundgrowth rate in areal den- 
sity would have been approximately 24 percent. If 
we restrict attention to linear density gains, the mod- 
eling shows that progress in signal processing and 
coding technology alone has accounted for almost 
a quadrupling of the linear density achievable 
with a ! 'typical" set of recording components. 

As communications theorists, we define the "chan- 
nel" as the part of the system over which we have 
no control. As such, we assume that the channel 
is fixed. Although read/write engineers for stor- 
age systems may not have direct control over the 
choice of the channel, the systems that make up 
the channel areconstantly being improved and these 
improvements have been the principle reason for 
the growth in information density in Fig. 1. Thus, 
instead of utilizing modern modulation and cod- 
ingsystems thatyield performancecloser to the chan- 
nel capacity, the designers of these systems have 
taken the alternative (and probably harder) approach 
of increasing the channel capacity itself. 

As a matter of fact, the modulation and cod- 
ing systems used in most of today's products do 
not differ markedly from the modulation and 
coding systems used in products a decade ago. As we 
shall see, themodulation and codingsystemsin these 
products were chosen to match a particular type 
of detector, called a peak detector, which has 
been an integral part of the system. The peak 
detector has seen a remarkable life. However, we 
maybe at a turning point, where the peak detec- 
tor and its associated modulation/coding/signal pro- 
cessing systems could be replaced by a sampling 
detector and an entirely new type of modula- 
tion/coding/ signal processing system. The new 
system looks very much like the systems that 
communications engineers are accustomed to 
seeing in advanced communications systems: par- 
ti i] i msi f-qu ligation, t I code \ herb d i 
tion, adaptive digital filtering, and so on. Both 
the old and new systems will be described in this 

Although many different types of media can 
be utilized for the storage of digital data, we con- 
centrate in this paper on systems based upon tra- 
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■ Figure 1 . Storage density versus time 

ditional magnetic recording. The techniques 
described, however, have applications mother types 
ofstorage systems such as those usingmagne-tic-optic 
and optical recording. 

One important difference between communi- 
cation systems and storage systems is the require- 
menton decoded error rate. Often in communication 
systems, the goal is a user error rate of ID 5 or 
I0" fi . Storage systems, however, often require 
error rates of 10 !2 or better. Furthermore, imple- 
mentations of error recovery procedures and 
their impact on the performance measures ol 
storage devices have often mandated that there 
be astrict requirement on the "raw" error rate at the 
output of the channel before the error correction 
decoder. 

Channel Model 

rhe "guts" of a magnetic recording system are: 
the write head, the magnetic medium, and 
the read head. (The write head could be the same 
as the read head and usually has been for disk drives. ) 
The write head is driven by a current source that car- 
ries the information to be stored. The write head 
radiates flux, which changes the state of magneti- 
zation of the magnetic medium immediately 
under the head. Actually, since the head is mov- 
ing with respect to the magnetic medium, any 
point on the magnetic medium retains the state 
of magnetization corresponding to the last flux it 
experienced from the write head as the head 
moves away from that point. 

On a rigid disk, the disk moves in acircular motion 
under the head. Information is stored on the disk 
in concentric tracks, the width of a track roughly 
being governed by the size of the write head. The 
density of recording is then the product of the 
number of tracks per inch (tpi) and the linear 
density of information along a track measured in bits 
per inch (bpi). Typical numbers for today's high 
end (i.e., expensive) rigid disk drives are: 3,000 
tpi and 30,000 bpi. 

There are at least two types of magnetic tape 
systems. In the first type, the head (or heads) remains 
stationary while the tape is puiled over the head. 
In the second, called a rotary head system, the 
head (or heads) is fastened to a spinning drum 
while the tape is moved slowly past the drum. 
This type of system is used in videocassette 



recurdingandutherappliratioiiswhere larger band- 
width isrequired.The head-to-tapespeed in the first 
type is governed by the speed of the moving tape, 
while in the second type, the head-to-tape speed 
is mostly a function of the rotations! speed of the 
drum and not the speed of the tape past the 
drum. In multiple write head tape systems, infor- 
mation usually is written simultaneously on many 
tracks, while in rigid disk storage systems, a single 
head almost always writes information on a single 
track. Rigid disk systems usually have a single 
head for both writing and reading (or, as in some 
recently announced products, write head physi- 
cally coupled with a read head), while tape sys- 
tems may have aseparate read head so that the system 
can read what is being written. 

The current into the write head induces a 
magnetization pattern on the track immediately 
below the write head. When a track is to be read, 
a read head is positioned over the track. Then, 
the magnetization pattern "frozen" on that track 
radiates flux that is sensed, or "read," by the read 
head. The read head produces avoltage that is symp- 
tomatic of the magnetization on the trackbeing read. 
There are primarily two types of read heads: 
inductive heads which contain coils of very fine 
wire and which produce a voltage proportional to 
the time derivative of the flux that passes through 
its coils, and magneto-resistive (MR) heads which 
produce a voltage directly proportional to the 
flux sensed by the head. MR heads produce larg- 
er read voltages lhan inductive heads, bul have a lim- 
ited dynamic range for linear operation. Only 
inductive heads have been used for writing, to 
this date. 

In rigid disk systems there is a separation between 
the head and the diskcalled the air bearing. This sep- 
aration is extremely small and the slightest imper- 
fection on thesurfacc of the disk (or any contaminant 
in the air bearing) could cause the head to "crash," 
Tape systems have the head (or heads) in contact 
with the magnetic surface. Because of the roughness 
of the tape, there is actually a nonzero average 
separation between the head (or heads) and the tape. 

Themagneticrecordingchannel is inherently non- 
linear because of the hysteresis that affects all 
magnetic media. However, magnetic recording 
specialists have found a way of linearizing the 
channel for a restricted range of inputs. They con- 
strain the current into the write head to take on 
only two possible values, for example, +A and -A, 
where the amplitude A is chosen sufficiently large 
so as to completely magnetize the magnetic stor- 
age medium in one of two directions. Thus, the 
hysteresis effect can be ignored. This type of 
recording is called saturation recording, and all 
practical digital storage devices use this approach. 
After information has been written on a track 
using saturation recording, the magnetic medium 
on that track would be alternately magnetized 
along the track, either in the direction of rotation 
or the opposite direction. A two-level write current 
waveform and its corresponding magnetization 
pattern on a track are shown schematically in 
Fig. 2. 

Since the write current can only take on the 
values +A and -A, the stored information is rep- 

Usually, one assumes that the time interval 
assigned to each channel bit is fixed. We call the 



The magnet- 
ic recording 
channel is 
inherently 
nonlinear 
because of 
the hysteresis 
that affects 
all magnetic 
media. 



IEEE Communications Magazine • December 1991 



SO 



A major 
problem 
affecting the 
read signal is 
the time- 
varying 
gain of the 
channel 



(b) Magnetization on Track 
■ Figure 2. Saturation recording 




m Figure 3. NRZI recording 



duration of a channel bit In oneconvention calied 
NRZI modulation, a channel bit equal to " 1 " is writ- 
ten as a transition in the write current (from +A 
to -A or vice versa) in the middle of a bit cell, 
and a channel bit equal to 0 is written as no tran- 
sition in the write current (i.e., the write current 
in the present bit cell remains at the same level 
as at the end of the previous bit cell). A repre- 
sentative sequence of channel bits and the corre- 
sponding write current after NRZI modulation 
are shown in Fig. 3. 

Assume that the write current is initially equal 
to +A and that transitions in the write current 
occur at times jiT^j^ ...JiT h where/,,/2, 
are integers such that y*j </ 2 < ... < /,-. Let u(t) be 
a unit step that occurs at ; = 0, 



mO- 



Hkd the write Current, /((), can be written a; 
/(;) = A + 2/l£{-l) i U (r-j f r iJ ). 



Ijct g (t) denote the read voltage corresponding 
to an isolated positive-going transition of the 
write current (from -A to +A) occurring at time t 
= 0. Then, if the channel werelinear; the output volt- 
age, V(t\ corresponding to the magnetization induced 
by the write current /(f) would be given as [1]: 

V(/)=.Ai(-iy«{<-y,r 6 ). 



A plot of a typical g(() taken from a thin film 
disk with thin film (inductive) heads is shown in 
Fig. 4. 

A common mathematical riiodel for such an 
isolated transition response is the Lorentzian 
pulse shape given by the formula: 




■ Figure 4. Isolated transition response from a 
thin film disk and thin film lead 

response at 50 percent of the maximum ampli- 
tude. The advantage of using this admittedly 
imperfect model is that it portrays the shape of 
the isolated transition response reasonably well, and 
requires only the single parameter PW ia . 

The preceding discussion assumes the validity 
of using linear superposition of the isolated tran- 
sition responses to form the composite output 
due to many transitions. This is, again, only a 
rough approximation to what actually occurs because 
of the interaction of the transition to be written 
with those that have already been written. The 
new transition couid be very close to the previous 
one, and the magnetic flux" radiated" from the medi- 
um where the previous transition was written 
influences what currently is being written. This 
flux, called the demagnetization field, pulls the tran- 
sition to be written closer to the previous transi- 
tion. We will neglect nonlinear effects in this 
paper, but one must bear in mind that these can- 
not be completely ignored in the practical design 
of high density systems. 

Another major problem affecting the read signal 
is the 1 time-varying gain of the channel. This can 
be due to many phenomena, including variation 
in the spacing between the read head and the 
medium over time, or the presence of physical defects 
in the medium itself. In disk systems, a map of 
each disk surface is usually made, identifying por- 
tions of the disk containing defects so that these por- 
tions will not be used. 

So far we have not discussed the effects of 
noise. It is common to assume that the noise in 
the system is additive and Gaussian . Usually, it is also 
assumed that the noise consists of two compo- 
nents: a Gaussian white noise component due to 
the electronics on the read (i.e., receiver) side, 
and a Gaussian colored componen t due to the medi- 
um. The spectral characteristics of this colored noise 
are essentially the same as would be obtained 
from passing white noise through the linear trans- 
fer function characterizing the system. More 
complicated models for the noise exist, particu- 
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larly for ihin film media, where it has been shown, 
for example, that there is more noise at the cen- 
ter of a transition than where there is no record- 
ed transition. A plot of the variance of this noise 
as a function of position within an isolated transition 
response for a representative thin film medium is 
shown in Fig, 5. This signal-dependent noise 
leads to opportunities for improved detectors, 
but this advanced subject is still exploratory and 
lies beyond the scope of this paper. 

Peak Detection Systems 

T" he design of the modulation, coding and sig- 
I nal processing in past magnetic recording 
products has been driven by the detector chosen 
to detect the transitions in the channel input 
waveform. This detector, called a peak detector 
[2], has the advantage of being both robust and 
extremely simple to implement. However, by its very 
nature, it works best at low linear densities. A 
block diagram of a typical peak detector is shown 
in Figure 6. 

There are two paths through the detector. The 
top path is used to "qualify" a peak, i.e., to ensure 
that the peak has sufficient amplitude. This path con- 
sists of a linear filter, ,(/), a full wave rectifier 
(FWR), and a threshold testing circuit. The bot- 
tom path is used to locate the peak by differenti- 
ating the signal after the linear filter H 2 (f) and 
then passing the differentiated signal through a zero- 
crossing detector. The detector only accepts a 
peak if the peak amplitude was large enough to 
pass the qualification test. 

Once a peak is detected by the peak detector, 
it is thought to be due to a transition in the input 
waveform. A device called a phase-lock loop 
(PLL) is used to derive timing from the position 
of the detected peaks. The PLL produces a clock 
of period T b seconds by which to identify channel 
bit intervals (sometimes called "bit cells"). Then, 
if an output pulse is located in a bit interval, that 
bit interval is said to contain a transition. Using 




■ Figure 6. Block diagram of a peak detector 



the NRZ1 precoding convention of Fig. 3, a bit inter- 
val with a transition corresponds to a recorded 
"1", and an interval without corresponds to a "0." 

Note that the use of the NRZ1 prccoder 
ensures that the reconstruction of the recorded 
sequence is insensitive to polarity inversion of the 
channel output waveform: a peak, regardless of 
its polarity, corresponds to a "1", and the absence 
of a peak corresponds to a "0." From another 
perspective, if the detector had to recover the 
actual recorded write current from the corre- 
spondence between peaks and write current tran- 
sitions, a single error from a "missed" peak would 
propagate until the next "missed" peak. 

The output of the peakdetector is used as an input 
to the PLL, and the output clock produced by the 
PLL is constantly being adjusted so that the aver- 
age peak position is centered with respect to the edges 
of the bit interval. 

If one examines the waveform produced by 
the linearsuperpositionoftwoLorentzian pulses (of 
opposite sign)separated by cePH^pseconds, one finds 
that this waveform will contain two peaks sepa- 
rated by f)PW 5a seconds, where ft > a. The param- 
eters a and £ are related by the formula 



Once a peak 
is detected by 
the peak 
detector, it is 
thought to be 
due to a 
transition in 
the input 
waveform. 



which for small a becomes 



For a much greater than 1, /3 is approximately 
equal to a, but as a approaches zero, /3 approach- 
es a fixed, limiting distance given by the value jf . 
Thus, the peaks will be centered in their bit inter- 
val only at low densities. Figure 7 shows a plot of 
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a Figure 8. Roll-off cum for Lorentzian transition 
response 

the normalized peak separation jj as a function of 
normalized transition spacing a. 

Of course, another effect of the interference 
of pulses is the decrease of the peak amplitude. 
The amplitude of peaks in the superposition of 
an infinite sequence of Lorentzian pulses (of 
alternating polarity) as a function of the transi- 
tion separation '/' = aPW^ seconds (sometimes called 
the "roll-off curve") also has a simple closed- 
form solution: 



Y(cx) = 



(una) 



where sinhfc) = (e?-e-*)!2 is the hyperbolicsmefunc- 
tion. (For fans of complex analysis, wc remark 
that one approach to deriving this formula pro- 
vides a pleasant exercise in contour integration.) Fig- 



recording system with peak detection 
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ure 8 shows the rolloff curve as a function of nor- 
malized transition frequency. The amplitude i/(a) 
also plays a roie in determining the design density 
as limited by the amplitude qualification of peaks 
in the detector. 

To improve the performance of the system at 
moderate information densities, two techniques are 
used, often in combination. These techniques, "equal- 
ization" and "modulation coding," are now briefly 
described. 

Equalization refers to the use of a linear filter 
at the output which changes the overall impulse 
response of the system before the peak detector. This 
equalizer is often called a pulse slimmer [3], its 
purpose being to create a channel whose isolated 
transition response is a thinner pulse than that 
produced by the unequalized channel. Many dif- 
ferent equalization techniques arc utilized, but 
they all have the effect of increasing the noise at 
the equalizer output since pulse slimming is achieved 
by boosting the high frequencies where there is 
little signal and much noise. We will not devote much 
space in this paper to the discussion of equaliza- 
tion. However, we will mention a new class of tar- 
get transfer functions derived from partial-response 
signaling, a method familiar to many communica- 
tions engineers. 

A second performance enhancement tech- 
nique for peak detectionchannels is the use of a mod- 
ulation code, or, more specifically, a special class 
of these codes called run length-limited (RLL) or 
{d, k) codes. Here d and k are nonnegative inte- 
gers with k strictly larger than d. A (rf, *)-cncoded 
sequence musl satisfy the constraint that symbols 
"1" must be separated by at leasts and at most A sym- 
bols "0." (Tbechoice of the lettered andA: todescribe 
thesecodes is unfortunate because of their different 
meaning in the discussion of error-correction codes, 
but several attempts to change the notation have met 
with little success. Fortunately, in this paper we 
will make only brief remarksabout error-correction 
codes, so no such confusion should arise). 

We willdescribe (d, k) codes in terms of the restric- 
tions that these codes produce on the write cur- 
rent. Whereas, for the uncoded case, the minimum 
time intervalbetween transitions in thewrite current 
can be as small as Tt, and the maximum time inter- 
, il i csween atlja > n tr; nations coul i be infinite, 
when a id, k) code is utilized, the minimum time inter- 
val between transitions is (d + 1)T$ and the maximum 
time interval between transitions is (k + i)Tt,. 
Since the magnetic medium is moving past the 
head at some velocity, v, the minimum spacingbetween 
transitions on the medium is v(rf+l)r fr . The value 
v(d+ l)r fc is referred to as the linear transition 
density, and if this distance is measured in inches, the 
linear density is measured in flux changes per inch 
(fci). Typical high-end magnetic disk systems 
today operate at between 20,000 and 30,000 fci, 
and magnetic tape systems operate at linear den- 
sities as high as 62,000 fci. 

The necessity for the parameter £ to be finite 
follows from the fact that the PLL requires feed- 
back from the peak detector in order to position 
the bit interval boundaries. If there is a large time 
interval between transitions, there will be a corre- 
sponding large time interval during which the 
PLLsees nocorrective signal. Such a situation would 
result in an undesirable drift in the clock signal 
produced by the PLL. One might think that a data 
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scrambler could be used to make the likelihood of 
such an event very small. However, diskdesigners are 
interested in worst case behavior: once a file writ- 
ten on a disk can no longer be read reliably, there 
is no second chance to retransmit the file. 

A block diagram of a typical recording system 
using a peak detector, linear equalization, a (d, k) 
modulation code, and an error-correcting code is 
shown in Fig. 9. 

The user data (possibiy compressed) is first passed 
through an encoder for an error-correction code 
(ECC) such as for a Reed-Solomon code. ([4].) 
The output of the ECC encoder is then encoded again 
using a (d, k) modulation encoder. Finally, the 
output of the (d, k) encoder is NRZI modulated 
by the signal generator, forming a write current of 
two levels, where a transition in thewrite curren (cor- 
responds to a "1" in the (d, £)-cncoded stream. 
On the read side, the output voltage waveform 
from the read head is equalized and passed 
through a peak detector(theever-present phase-lock 
loop is not shown), where bit cells containing 
detected peaks are converted to Is and bit cells 
without peaks are converted toOs.Thecorresponding 
binary stream from the peak detector is next 
passed through the (d, k) decoder, then through 
the ECC decoder (and decompressed, if appropri- 
ate). Notice that the NRZI precoding convention 
provides a very elegant way to translate the constraints 
imposed uponthercadbacksignalbythepeakdelec- 
lion system into constraints on the sequences to 
be generated by the modulation encoder. 

M osl comm imicat ion engineers havesome famil- 
iarity with data compression and ECC schemes, 
but not as much with modulation coding techniques, 
particularly those used in data storage, The 
remaindcrof this papcrwill be devoted to a discussion 
of such modulation codes, beginning with {d, k) 

(d, k) Codes 

/n this section we describe some of the (d, k) 
codes that have been used in commercial stor- 
age products, Further details may be found in a vari- 
ety of references [5-8]. "We begin with a very 
simple example of a {d, k) code where d = 0 and 
k = 2. This code, sometimes called Group Code 
Recording (GCR) [9], has found application in mag- 
netic tape drives. The code description is shown 
i n Table 1. 

User bits are encoded, four bits at a time, into 
five-bit code words. It is easily verified that any 
concatenation of these five-bit code words satis- 
fies the (0,2) constraint. 

The rate of the GCR code is 4/5 and one might 
wonder what is the maximum code rate possible 
for any (0,2) code.The answer to four decimal places 
is .8792. For (d,k) codes in general, the maximum 
code rate (sometimes called the capacity of the code) 
is given by the base-2 logarithm of the largest real 
root of one of the following equations, depending 
on whether k is finite or not: 




The purpose for choosing the parameter d to 
be strictly greater than zero is to increase the 
information density along a track while keeping 
the time interval between adjacent transitions greater 
than some fixed constant. Assume that T mi „ is the 
smallest time interval that can be allowed between 
neighboring transitions in the two-level write cur- 
rent. If d is equal to zero, then we must choose T b 
such that T b > T„, t „. For this choice of T b , since 
onecoded binary digit cannot contain more than one 
informatiojibit, the maximum information fate that 
can be supported by the two-level input wave- 
form would be VT b < \/T,„ in bits/second. Now assume 
that d is chosen as an integer strictly greater than 
zero. Since the minimum time intervalbetween tran- 
sitions is now (d + l)7t„ T b can he chosen to equal 
T m J{d + 1) and the coded binary digits then 
occur at a rate of VT b = (d + \)IT mia binary dig- 
its/second. Fora fixed value of 7"„„„, this corresponds 
to a coded symbol rate that is a factor of (d + 1) 
times the transition rate. Unfortunately, this increase 
is not in the information rate but in the coded 
binary symbol rate, and the number of coded 
binary digits required to represent one informa- 
tion bit generally increases as d does. Let R be 
the ratio of the number of information bits to 
coded binary digits for a given (d,k) code. Then 
the information rate for this system using the (d, 
k) code i%R!T b = R(d + \)IT mitt bits/second. The prod- 
uct R{d + 1 ), called the density ratio of the code 
[10], represents the increase (if R{d+\) > I) or 
decrease (if R(d + 1) < 1) in information rate 
using a (d,k) code as compared to an uncoded 
system. We are particularly interested in systems 
where the density ratio is strictly greater than 1, 
for then we are storing information at a higher 
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■ Table 3. Parameters of commonly used (d,k) 
codes 

rate than the highest possible transition rate allowed 
in the two-level input waveform. The most com- 
mon (d,k) codes used in past and present prod- 
ucts are listed in Table 3. 

Wehave already discussed the (0,2) (GCR) code. 
The (1,3) code has many names, including Miller 
code, Delay Modulation, and Modified Frequen- 
cy Modulation (MFM) code [9], It is a rate 1/2 
code where one information binary digit is encod- 
ed into two coded binary digits. It is asystematiccode 

in that the information sequence i h ij j" t ... 

gets transformed into the coded sequence ij,i : , 
<2> *2. ■ ■ ■ 'k. z \ There is a simple rule for insert- 
ing the extra digits {z k }: Choose z k to be 0 unless 
it is to be inserted between two information 0s, in 
which case choose z k to be a 1. 

We will now demonstrate how to derive the 
coding rule for the (1,3) code. The derivation 
illustrates some of the basic ideas involved in the 
design of constrained recording codes. However, 
more powerful techniques are required in gener- 
al, as will be discussed in more detail. We begin 
with a finite state transition diagram (FSTD) 
which produces all binary sequences satisfying 
the (1,3) constraint. This FSTD is shown in Fig. 
10. Constrained code sequences are generated by 
taking walks on the graph, following the arrows 
a nd reading off the code symbols that label the edges 
traversed. 

The capacity for a (1,3) code is C~ .5515, and 
we desire a code rate close to C, such as a rate R 
= 1/2. Thus, we seek a graph that can represent a 
rate 1/2 encoder finite-state-machine, namely, 
one with two edges emanating from each state 
(one for encoding a 0, the other for encoding a 1) 
and with binary code words of length 2on each edge. 
In order to obtain agraph with edges labeled by code 
sequences of length 2, we form flic "second power 
ofthcFSTDofFig. 10," that is, the FSTD obtained 

i Kin tcps« 2 in Fig, I tphi sli i 

in Fig. 11. 

Note that state 3 in this graph is deficient in 
that it has only one edge emanating from it. How- 
ever, we are very fortunate in that the only way to 
enter state 3 is from state 1, and state 1 has 3 
edges emanating from it. Eliminating the edge 
that goes from state 1 to state 3 eliminates state 3 
from considcrationaltogether.resultingiathe FSTD 
shown in Fig. 12. 

We now note that states 1 and2 are "equivalent," 
meaning that both states produce the same set of 
code sequences. Thus, these states can be combined 
into one state as shown in Fig. 13 [8]. 

Finally, we obtain the state diagram for a rate 
R = 1/2 encoder for a (1 ,3) code by labeling the edges 
in the form a/bc, where a is the information bit 
that is the input to the encoder and be is the pair 
of (1,3) constrained binary digits produced by the 
encoder. This isshown in Fig, 1 4.This encoder is pre- 
cisely the same encoder as used in the Miller 
(1,3) code. 




■ Figure 1 1 . Two-step transition diagram for (1,3) 
constrained binary sequences 




The rate 2/3 (1,7) code is arguably the most 
popular (d,k) code in use today. Several varia- 
tions of thiscode exist [1 1-13]. Asimple and elegant 
description, due to Jacoby, begins with the encod- 
ing table in Table 4. 

If the encoder is based solely upon a table lookup, 
however, we note that theuser datasequences 00.00, 
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and no code word begins with more than four Os 
or ends in tnoie than three Us). It can also be ver- 
ified that every information sequence can be decom- 
posed uniquely into a sequence of variable length 
strings in the left-hand column of the table. In 
addition, the variable length code words consti- 
tute a prefix-free code (no code word is a prefix 
of another code word), so that unique decoding 
of code sequences can be accomplished. One can 
also describe a sliding-block decoder for this code in 
such a way that decoding errors due to a single 
code symbol in error cannot affect more than 
four user bits. 

Prior to a few years ago, no general theory 
existed for the design of modulation codes (such 
as (d,k) codes) with minimum code word length, 
rinke-staleencoders,ands!iding-blockdecodcrs,at 
rates arbitrarily close to capacity (or equal to 
capacity, when the capacity is a rational number). 
Now, however, there is systematic technique forcode 
construction [8, 14]. The method, called the slid- 
ing-block code algorithm, allows for the design of 
a practical, efficient (d,k) code for any choice of 
the parameters d and k at any rational rate up to 



I Figure 1 5, Sliding block decoder 



00.0 1, 10.00, and 10.01 produce channel sequences 
(namely, 101.101, 101.100, 001.101, and 001.100, 

pecti i hi mh ^ ili ( onstraiat 
The "substitution table" in Table 5 is used to cor- 
rect for these violations. 

When the encoder is ready to encode a pair of 
user data bits, it "looks ahead" to the next pair of 
user data bits to see if using Table 4 for both 
pairs would result in a violation of the (1 ,7) constraint, 
if no violation would occur, the encoder uses 
Table 4 to encode the first pair. If a violation 
would occur, the encoder encodes these two pairs 
using Table5. 

Decoding can be accomplished in a state- 
dependenl manner using a "slidiug-biouk decoder/' 
a generalization of the code word "table look-up" 
decoder used for the (0,2) and the (1,3). A 
schematic ofusliding-biuck decoder is shown in Fig. 
15. 

As illustrated for a rale m/n code, the decoding 
of a code word of length n depends on the con- 
tents of a decoderwindow that com ains I lie code word 
in question, as well as a fixed number of past and 
future codewords ("lookback" and "look-ahead"). 
In the case of the ( 1,7) code, the decoder decodes the 
current three-bit code word by looking ahead at 
the next two upcoming code words. In this way. a 
single incorrectly detected code symbol can prop- 
agate into a burst of at most six user bits (in fact, 
the burst length does not exceed five user bits). 

Another modulation code used in several disk 
drive products is a rate 1/2 (2,7) code. One encod- 
ing and decoding table for such a code, invented 
by P. Franaszek, is given in Table 6. 

It is easily seen that the code rate is 1/2 (every 
code word contains exactly twice as many binarydig- 
its as the information sequence it represents) and 
that any concatenation of the variable length 
code words satisfies the (2,7) constraint (each 1 
in every code word is followed by at least two 0s, 




0100 

1000 

000100 

100100 

001000 

001 001 00 

00001 000 



The sliding 
block algo- 
rithm was 
first used to 
find a five- 
state encoder 
fora(lj) 
code. 



tela 

the capacity C, specifically for any integers m and 
n satisfying m/n < C, the algorithm yields a finite- 
state encoder that accepts m binary inputs and 
generates/; binary outputs, and a state-independent 
decoder requiring only finite look-ahead and 
look-back, thereby limiting error propagation. 

The sliding-blockalgorithmwas first used to find 
a five-state encoder for a(l,7) code. It converts 
two binary information digits into three coded 
binary digits [12], and has been shown to gener- 
ate the same set of code sequences as the (1,7) 
code described earlier. More recently, another 
rate 2/3 encoder with only four slates [13], the 
t possible for a rate 2/3, (1,7) code f 15], 
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The nature of 
sliding-block 
codes is to 
propagate 
errors at the 
decoder 
input into a 
finite burst 
of decoded 
data errors. 



was invented using the hiding-block code algorithm. 

The nature of liding-biocke t o[ t 
errors at the decoder input into afinite burst of decod- 
ed data errors. Therefore, random-burst correct- 
ing ECC have been applied as a sort of add-on to 
handle these rare, short burst errors. Single burst 
error correction codes and some specially crafted 
codes similar to Fire codes were used in some 
products, for example. Now, this design philoso- 
phy is slowly changing as engineers come to real- 
ize that error correction codes must be an integral 
part of any storage system and can lead toboth high- 
er reliability and storage efficiency. Today, the 
most general class of codes used are Reed- 
Solomon codes, although most products using Reed- 
Solomon codes are still not very aggi i v t i 
theii rrcction 1 sol ytccrror 

correcting codes). Recently, however, several 
general purpose chips have become available [16, 
17] that can correcl (on the fly) many byte errors 
(e.g., 10) in a code word at speeds in excess of 
what is required for today's products. The opti- 
mization of the recording system, incorporating 
advanced modulation, detection, recording codes, 
and ECC is a challenging problem, beyond the scope 
of this paper, but it seems quite reasonable to 
conjecture that the newly available ECC power 
will find its way into future storage devices. 

Partial Response Systems 

n cccntly, researchers from IBM laboratories 
fl reported the results of an experiment demon- 
strating that an areal density of 1 gigabit per 
square inch could be achieved for the storage and 
reliable retrieval of digital data on a hard disk 
system [18]. This many-fold increase in density 
was achieved usinganumber of advanced techniques. 
One of these techniqu is a diffe it approach 
to combatting intersymbo! interference, some- 
times referred to as PRML, using partial-response 
(PR) signaling with maximum-likelihood (ML) 
sequence detection. 

instead of keeping the transitions far apart ■ 
using (d,k) codes, PRML allows the transitions to 
be close together, and the read signal, with its 
re ul 1 rig in', rsyml > im rl'erencc if qualized 1 i 
frequcncyrespon.se known as a ciass-4 partial-response 
channel [19]. The equalized signal is then detected 
by a maximum likelihood sequence estimator, i.e., 
a Viterbi detector [20], In this section we will give a 
brief summary of the PR and ML components of 
this system as they apply to magnetic recording. 

Partial Response Equalization 

We now turn to a more detailed description of 
partial-response signaling. Consider the two-level 
write current, /(t), shown in Fig. 16(a), where it 
should be noticed that the bit cell boundaries are 
now located such that the transitions occur at the 
edge ofa bit cell. Shown in Fig. 16(b) is the elementary 
pulse p(i). 

Note that the write current /(/) can be written 
in terms of the elementary pulse p(t) as: 

<w=£<w>c-«i) 

where a,- takes on the values +A or -A. Suppose 
that the channel is equalized so that the response 




■ Figure 16. (a) Write current, (b) Elementary 
pulse p(t) 

to the pulsep(0 is a signal A(r), referred so as the par- 
tial response signal. Then the readback voltage, V(t), 
is of the form: 

V<f)=2>;«'-'7i). 

Furthermore, assume that when h(t) is sam- 
pled at bit interval boundaries every T b seconds, 
the oniy nonzero samples are: h 0 = ft(0), h } - 
h(T b ), ... , li L = h{LT b ), The nonzero samples 
can be conveniently represented by the "partial- 
response polynomial" h(D) - Hq + Jtj D + ... + 
h L D L , where the factor D' signifies a delay of i 
time units T b . (In other words, h{D) is the D- 
transform of the sampled pulse response.) Thej- 
th sample of the readback voltage V{jT b ) can be 
written as: 



V(jT„) 



- i 



HhUT b -iT b ). 



V(jT„)- 

For magnetic recording systems with PW^T b 
approximately equal to2, comparatively little equal- 
ization is required to force the equalized channel 
to match a class-4 partial-response (PR4) channel 
where h(D) = 1 - D 2 [21]. At higher recording 
densities, one can choose a partial-response poly- 
nomial of the form h(D) = (1 - £>)(1 + D)" where 
n is chosen as f. positive integer greater than 1, to 
produce a response h{t) that is a better match to 
thechanncl pulse response [22]. ForPWV 7 * approx- 
imately equal to 2.25, the properchoice of n is 2, lead- 
ing to the so-called EPR4 (i.e., extended class-4 
partial-response) channel with h(D) = \+D-D 
- D 3 [22]. Eye diagrams for PR4 and EPR4 wave- 
forms are shown in Figs. 17 and 18, respectively. 
These diagrams [19] represent the overlaying of 
the channel output signal seen in each time inter- 
val 7'j,, assuming a random binary input sequence. 
One can clearly sec the nominal three-level 
(respectively, five-level) set of values at the sam- 
ple times for the PR4 (respectively, EPR4) 
response. The eye diagrams provide some useful, 
qualitative indication of the robustness of the 
sample values at bit cell boundaries in the pres- 
ence of additive noise and timing jitter. 

Although PR equalization in communications 
systems and, as will be discussed, in magneticrecord- 
ing systems is primarily used in conjunction with 
detection schemes that process samples of the chan- 
nel output waveform, it is interesting to note that 
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it has also found application in recording systems 
employing peak detection. For example, raised- 
cosine and cosine-squared filters (corresponding 
to PR polynomials A(D) = 1+0 and/i(D) = (l+D) 2 ) 
have long been used to achieve high frequency noise 
reduction and some degree of pulse slimming, 
although the realizations have typically not been min- 
imum-bandwidth. In addition, more recently it 
has been demonstrated that equalizing the chan- 
nel to an extended PR with polynomial of the 
form h(D) = (1 - D)(l +D)» provides perfor- 
mance improvement in peak detection systems 
[23, 24]. 

In a later section, we will describe a newly pro- 
posed coding method designed for a peak detec- 
tion system employing EPR4 equalization. 

PRML 

The applicability of partial-response signaling as 
a means of coping with intersymbol-interference 
in magnetic recording channels was suggested 
over 20years ago [21], and the use of a Viterbi detcc- 
torfor maximum-likelihood sequence estimation in 
the storage context was proposed almost simulta- 
neously with similar proposals for data communi- 
cations [25, 26], 

During the past I Oyears, additional analysis, sim- 
ulation and, finally, laboratory experimentation con- 
firmed the potential value of the PRML system, 
[27-30]. In both simulation and experiments, the 
benefits in linear density that can be obtained 
over systems using RLL-coded peak detection 
have been found to be approximately 30 percent 
[31]. In addition, further research results indicat- 
ed that the digital nature of the signal processing 
in PRML leads to advantages in electronic imple- 
mentation (particularly VLS I) and extendibility, e.g., 
via coded-modulation [32, 33], digital adaptive equal- 
ization [34], and digital timing and gain control 
[30]. 

This activity has finally culminated in the 
incorporation of PRML into magnetic tape prod- 
ucts, [35] and, very recently, magnetic diskproducts, 
[29]. 

We new briefly review the principles of Viterbi 
detection for combatting intersymbol interfer- 
ence, particularly as these concepts relate to cod- 
ing for recording channels based upon 
partial-response. Recall that, in the NRZI (1/(1 e 
£>)) precoded channel, a code symbol 1 produces 
a positive or negative transition in the write cur- 
rent (respectively, a positive or negative pulse at 
the channel output) and a code symbol 0 produces 
no transition (respectively, pulse). This correspon- 
dence translated the restricti m on minim i n »nd 
maximum transition spacing(respectively, run lengths 
of time intervals with no pulses at the input to the 
peak detector) into the easily represented (d,k) 



Similarly, in the PRML setting, an Interleaved 
NRZI or INRZI (1/{1 ©£> 2 )) prccoder converts 
the constraints on the samples at the input to the 
detector into simply described constraints on the 

e it 4 jtotheinpt f tl r J i 
aswc will describe. In the INRZI-precoded PR4 
channel, a code symbol 0 at the input to the precoder 
will produce a sample 0 at the channel output, 
and a code symbol 1 will produce a sample value 
of either +1 or -1 Themaximu ikeliho j quen 
detector based upon the Viterbi algorithm takes 




■ Figure 1 8. Eye diagram for EPR4 channel 
a received (noisy) sample sequence of length n, 

y=y\$i y«> 

and determines in a recursive manner the chan- 
nel output sequence (and the corresponding data 
sequence) that provides the best fit in a least-squares 
sense to the observed sample sequence, as shown 
schematically in Fig. 19 for the INRZI-precoded 
I T nnel.(Th ulli< isiitc m ■ 1 1 hi 
thi v\ «d zjuonufMLdctectionisduetoGottfried 
Ungerboeck). A rough analogy to linear regres- 
sion can be made, where one fits a line to a set of 
observations in such a way as to minimize the 
sum of squared errors. 

The PR4 channel, with polynomial h(D) = 1 - 
D 2 , can be considered as two time-interleaved 
"dicode" partial-response channels, each with poly- 
nomial h{D) = 1 - D. A common method of 
applying the Viterbi algorithm to maximum-like- 
lihood sequence detection for the PR4 channel is 
to de-interleave the samples of the readback 
waveform to form two streams of samples, a 
stream made up of the samples at odd time indices 
and a stream made up of the samples at even 
times indices. Then Wo Viterbi detectors matched 
to the 1 -D channel can be used , one for each stream. 
In practice, one might even use just one detector 
in a pipelined fashion. This interleaving approach 
will be important when we come to discussing 
codes for this channel. 

For an NRZI-precoded 1 -D channel, the 
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channel input/output sequences arc conveniently 
represented by ihe trellis structure in Fig. 20. 

Within each stage of the trellis, shown in the lower 
portion of Fig. 20, each edge is labeled with an 
input bit (before the slash) and a channel output sym- 
bol (afterthe slash). Input/output sequence pairs are 
generated by reading off the edge labels as one 
follows a path through the trellis from left to 
right. The two states at each time correspond to 
the parity of the number of input Is so far, which 
determines the polarity of the channel output result- 
ing from the next input 1. Alternatively, the states 
can be thought of as the polarity of the write cur- 
rent at the end of the last bit cell. 

The Viterbi algorithm fits the time-indexed obser- 
vations at the channel output with two allowable 
channel output sequences: one minimizing the 
sum of squared errors over all possible noiseless out- 
put sequences ending in the first state of the trel- 
lis at time n; the other minimizing fhesum of squared 
errors over the noiseless output sequences ending 
in the second state of the trellis at time n. 

The recursive algorithm for frndingthese two "sur- 
vivor" sequences is based upon aconcep t from dynam- 
ic programming called the principle of optimality. 
What it says, and what can be readily checked, is 
that each of the two survivor sequences at any time 
n + 1 must correspond to an extension of one of 
the survivor sequences from lime n. The applica- 
tion of this principle to the 1 - D channel leads to 
several elegant interpretations and implementa- 
i >n tit j ii i ir >t i 1 - i tii ?• 

basedon the "difference mctric,"denoied£>M„ .which 
is simply the difference of the accumulated squared 
errors for the two survivor sequences at a specified 
time n. The recursive "difference metric algo- 
rithm/'first published in [36] (see also [30]), isdescribed 
in Fig. 21. Note that only three of the four possible 
survivor extensions can occur. 

This algorithm also can be interpreted as a 
"dynamic thresholding" scheme [37], as follows. Two 
threshold values, an upper threshold T% and a 
lower threshold T<„ , are initialized at time n = Q 
to +1/2 and -1/2. If the sample value at time n + 
l,y„ +! falls in the upper interval, y„+i > T", the 
first extension is selected, and the new thresholds 
are set to Tg+i =y„ + i and r'+i =y n+l - 1. lfy„ +i 
falls in the middle interval, T/, < y„ + i£ 7JJ, the 
second (parallel) extension is selected, and the thresh- 
olds remain unchanged, r„"+i =T„" and 7^+1 = 
T'„. Finally, if v„ +] falls in the lower interval, y„ + 1 
< Tj,, the third extension is selected, and the 
thresholds are set to T„" + , = y„ +] + land Tj, + i = 
y„ +) .SeeFig. 22. 

The first and third extension possibilities 
cause the survivor sequences to "merge" and, there- 
fore, in both survivor sequences, all decisions 
prior to the merge agree with those in the maximum- 
likelihood estimate that is ultimately generated. Note 
that, in the absence of noise, the merges take 
place precisely when the input bit is a 1 . 

The parallel extension option, on the other 
hand, defers the decision about the bit following 
the last merge until a future merge settles the mat- 
ter, Consequently, the detector must keep a record 
of the survivors (called the path memory or trellis 
history) at least back to the last merge. In practical 
applications, it is therefore very desirable to force 
frequent merges by limiting the separation of Is at 
theprecoder input, thereby reducing thelikelthood of 
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errors caused by truncation of the path memory to 
amanageable length. As mentioned earlier, in record- 
ing systems the preferred practice is to guarantee such 
aproperty of the recordedsequcnccs by meansof con- 
strained coding, rather than to achieve it only prob- 
abilisticallyviascrambliiig.This constraint on the input 
will play a role in the next section when we discuss 
the recording code constraints lor PRML channels. 

In general, if the partial-response polynomial 
is of degree L, and if the input to the channel is a 
binary sequence, then the Viterbi detectorwili require 
2L states. For example, the Viterbi detector for 
the EPR4 partial-response channel, where h(D) 
= 1 + D - D 2 + O 3 , uses an eight-state trellis, 
shown in Fig. 23. Here the trellis states represent the 
write-current levels, denoted 0 and 1, at the ends 
of the last three-bit cells. 

An important performance indicator for a 
trellis-based detector is the free squared-Euclidean 
distance which, roughlyspcaking, measures thesep- 
aration between thesequencesmost likely to becon- 
fuscd when corrupted by channel noise. The free 
distance can be derived from the trellis, as fol- 
lows. We consider the sum of the squared differences 
between the noiseless outputs for every pair of paths 
in the trellis that start in a common state and end 
in a common (but perhaps different) state. The min- 
imum of this sum is the free (squared-Euclidean) 
distance. For the 1 - D trellis shown in Fig. 20, 
the free squared-Euclidean distance is equal to 2. 

In a channel with additive white Gaussian 
noise, with zero mean and standard deviation o, 
the probability of an error event, for moderate to 
high SNR, is then well-approximated by: 



- event) 




in defining the free 



■ Figure 23. Eight-state trellis forEPR4 channel 



squared-Euclidean distance we do not insist that 
the ending state be the same for the two paths. Of 
course, the expression for the probability of error event 
must then be modified accordingly. With this defi- 
nition, for example, the free squared-Euclidean 
distance for the 1 - D trellis would be equal to 1. 

Codes for Partial Response 
Channels 

/n this section, we will describcsevcral coding tech- 
niques developed for use in recording systems 
using partial response equalization. We will first 
describe a code designed for a new peak detec- 
tion system incorporating EPR4 equalization. We 
then turn to a class of constrained codes, denoted (0, 
GIF) codes, that have been used in the implemen- 
tation of PRML channels in disk drives. These 
constraints incorporate the analogue of the k 
constraint in (d,k) codes, as well as a new con- 
straint that affects the required length of the 
Viterbi detector path memory. Finally, we turn to 
the consideration of codes intended to improve 
the noise immunity of partial-response channels, and 
we indicate several exploratory directionsproposed 
for trellis-coded modulation in storage channels. 

Codes for Peak Detection for the EPR4 
Channel 

This section describes an exploratory technique 
for extending the use of peak detection to the 
EPR4 channel [38]. Recall that the EPR4 chan- 
nel corresponds to the partial-response polyno- 
mial h(D) = 1 +D - D 2 - ffi. The eye pattern for 
this equalization (using the minimum bandwidth sig- 
nal having these sampled values) was given in 
Fig. 18. It should be noted from this figure that 
the channel output waveforms exhibit a number 
of different types of peaks with varying ampli- 
tudes and that, at the normal sampling point for 
the EPR4 channel, the waveforms can assume 
five different values. 

If one restricts the input waveform so that the cor- 
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responding binary sequence (using NRZI modu- 
lation) satisfies the { 1 ,7) constraint, the resulting eye 
pattern is as shown in Figure 24(a). 

Now all but the large peaks have been eliminated 
by the code, and a peak detector could be used to 
identify transitions in the input write signal. At 
the normal samplingpoint for the EPR4channel the 
waveform still has five levels. If the (1,7) coded 
sequence is first passed through a precodcr with 
transfer function ]/(l e D 2 ), the resulting eye 
pattern is as shown in Figure 24(b). Now only the 
smallest peaks are retained in this waveform, and 
once again a peak detector can be used to recov- 
er the input write current. At the normal sam- 



■ Figure 25. Response of£PR4 channel to differ- 




pling point fortheEPR4 waveform, thiscoded wave- 
form has only three levels. 

Figure 25 shows the response of the EPR4 
channel to a number of different input wave- 
forms. It is important to note that a pulse of 
duration 27 (here T is the sampling interval) 
results in a pair of peaks of heights +A and -A 
separated by approximately 2T, while a pulse of dura- 
tion T produces a pair of peaks of height approxi- 
mately +.6&4 and -6SA and separated again by almost 
2T. If the number JV of input transitions spaced T 
apart exceeds 2, the ouput waveform will contain 
two peaks of amplitude .68A spaced approxi- 
mately NT apart, as illustrated for N = 3, 4 at the 
bottom of Fig. 25. Tn general, the peaks of ampli- 
tude/! are out of ph e by 1/2 a it cell from the peaks 
ofamplitudc .6H4. When the basic waveforms shown 
in Fig. 25 are concatenated with each other, the shapes 
of the output waveforms remain approximately as 
shown in Fig. 25, provided that the last transition 
of the preceding basic waveform is no closer than 
3Tfrom the first transition of l he basic waveform 
that follows it. The basic waveforms producing large 
peaks need only be separated by two T or more. 

Using these observations as motivation, a code 
has been devised for use with this INRZI-precod- 
ed EPR4 channel (where the precoder has trans- 
fer function 1/(1 ® £> 2 )) such that a peak detector 
that can reliably delect peaks and discriminate 
betwcenlarge and small peakscan identify thebina- 
ry sequence that corresponds to the write current. 
The codcalsoproduces either a largeorasmall peak 
in the readback waveform every eight bit cells for 
timingrecovery.The maximum possible rate for this 
code is C = .8485, which is approximately 25 per- 
cent higher than for a (1,7) code. 

(O.G/I) Codes for the PRML Channel 

The gigabit per square inch experiment and the 
fir "niter Statesdisl drive product employingPR4 
equalization utilized code sequences satisfying a new 
type of constraint called a (0, 67/) constraint. 
This constraint imposes run length limitations 
that aid timing and gain recovery and simplify the 
design of the Viterbi detector for the channel. A 
block diagram of the PRML system using this 
code is shown in Fig. 26. The channel response is 
equalized to that of the PR4 system. As described 
previously, the PR4 Vi terbi detector is decomposed 
into a pair of interleaved detectors matched to 
the I -D channel. 

The constraints on the PR4 channel outputs 
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are twofold. First, for accurate timing and gain recov- 
ery, it is desirable to limit (he number of consecu- 
tive zero samples in the noiseless PR4 channel output 
sequences generated by the code sequences. Sec- 
ond, in order to minimize performance degradation 
resulting from the truncation of the path memory 
in the interleaved 1 - D Viterbi detectors, the 
maximum run length of zero samples in each of 
the two interleaves comprising a channel output 
sequence should be limited. 

As mentioned previously, the INRZl-precodod 
PR4channel generates asample Oat thechannel out- 
put for a code symbol 0 at the input, and a sample 
value of either +! or -1 at the output for a code 
symbol 1 at the input. Therefore, the first con- 
straint translates into a global constraint, denoted by 
the symbol G, on the maximum run length of 0 
symbols in any code string. This constraint is 
essentially the same as the k constraint in the (d, 
k) codes. The second constraint corresponds to an 
interleaved constraint, denoted by/, on the maximum 
run lengths of 0 symbols in each of the interleaves 
of a code sequence. The "0" in the notation (0,G//) 
can be treated as the analogue of the global d con- 



straint. Here, it serves to emphasize that there is 
no restriction on the minimum separation of 
nonzero samples at the channel output; that is, 
the code sequences are not forbidden to have 
adjacent Is. Figure 27 shows an example of a code 
sequence satisfying a (<), Gil) = (0, 4/4) con- 
straint, similar to the constraints used in the com- 
mercial PRMLchannel, alongwith thecorresponding 
noiseless channel output waveform, the global 
sample sequence, and the interleaved sample 
sequences. Note that adjacent nonzero samples 
can have the same polarity, that is, they do not 
correspond directly to peaks in the outputwaveform. 

As with the {d, k) constraints, there arc simple 
finite-state transition diagrams that describe the 
(0, Gil) constraints from which onecan compute capac- 
ities and construct sliding-block codes. The capaci- 
ties of several (0, Gil) constraints of practical interest, 
as well as the parameters of rate 8/9 codes satisfy- 
ing these constraints [39] are given in Table 7. 

J. Eggcnberger first discovered the optimalblock 
list of length 9for the (0, 4/4) and (0, 3/6) constraints, 
i.c„ the largest collection of nine-bit code words that 
satisfy the prescribed constraints when freely con- 
catenated. The {0, 4/4) optimal block list contains 
279 words, listed in decimal representation in 
Table 8. 



■ Table 8, Maximal list oflength-9 words for the (0,414) 
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For the (0, 3/6) constraint, Eggenbcrgcr found 
two optimal block lists containing 272 code words; 
the code words in one code being the time-reverse 
of the codewords in the other. For these constraints, 
specific rate 8/9 codes were obtainedby selecting 256 
code words from these optimal lists. (For more details 
about these codes and their logic implementa- 
tions, see [8, 39,40].) 

^rimprovir 

for the PR4 Channel 

The main reason for choosing the particular class 
of partial-response channels discussed previously 
is that the required equalizers donot boost the noise 
power excessively. The equalizer does change the 
spectral characteristics of the noise somewhat, 
but often this factor is ignored in the design and anal- 
ysis of the Viterbi detectors (when the slight loss 
in accuracy andperformance isdeemed acceptable). 

Asmentioned previously, the performance of the 
Viterbi detector at high signal-to-noise ratios (SNR) 
is known to be governed by the free squarcd- 
Euclidean distance of thecode, and the free squared- 
Euclidean distance for the 1 - D channel is equal 
to 2. Codes for improving the noise immunity for the 
PR4 channel involve eliminating some of the 
sequences generated by paths through the trellis 
shown in Fig. 20, with the objective of increasing 
the free squared-Euclidcan distance by a coding gain 
factory, where g > 1. In any coded system of this 
kind, one is trading information rate (represented 
by the code rate R) and complexity (of the 
encoder/decodn It git e li is the niodifieddetec- 
tor) for coding gain. The parameter Rg is called 
the asymptotic coding gain (ACG) far the coded sys- 
tem, and to a first approximation represents how 
much more (or less) noise can be tolerated by the 
coded system as compared with an uncoded sys- 
tem that yields the same error performance (at 
low error rates). 

The interleaved-FM code for the PR4 channel 
provides a simple, but representative, example of 
thistradeoffandthepotentialvalueofsimilarcoded- 
modulation schemes. The channel configuration 
is the same as in the previous section, with the 
PR4 channel decomposed into two interleaved 1 
-D channels, each preceded by a 1/(1 © D) precoder. 
The interleaved-FM code, as the name suggests, 
is defined by applyingseparatciy to the even and odd 
interleaves of the data string a simple rate 1/2 
code, often referred to as the FM (frequency 
modulation) code. The FM code is a block code with 



the 



rule: 



0-»01 



The interleaved-FM code therefore satisfies a (0, 
GIT) = (0, 2/1) constraint. We remark that the 
sequences produced at the precoder output com- 
prise the widely known biphase code, a simple 
rate 1/2 block code with code words 01 and 10. 
The precoded PR4 channel with the interleaved-FM 
code is therefore equivalent to the (unprecoded) 
PR4 channel with intcrlcavcd-biphase coding, 
and wewill use the names interchangeably. The trel- 
lis describing the output sequences as a function 
of the input data in the FM-coded 1 - D channel 
is shown in Fig. 28. If we use a pair of detectors in 
an interleaved fashion for detection of the inter- 
leaved-FM coded PR4 channel, as was done for 



Encoder: 1/2 (2:4) 

• Coding gain 4.8 dB 

d 2 M = 6 (|Olog 10 f = 4. 



■ Figure 28. Trellis structure for FM-coded 1 -D 




■ Figure 29. Minimum distance event for FM- 
coded I - D channel 

the uncoded PR4 channel, it suffices to examine this 
trellis. The minimum squared-Euclidean distance 
generated by an error-event is 6, so the distancegain 
factor is g = 3. An example of such a minimum 
distance event is depicted in Fig. 29. The ACG is 
therefore given by: 

ACG = 101og m Kg = lOlogin i-3 = I.BdR 

To assess the added hardware complexity asso- 
ciated with the trellis-coded scheme, one can use 
rough meaningfulmeasuresforlheencoder/decoder 
logic and Viterbi detector. Specifically, for the 
encoder/decoder, one might look at the number 
of states and the data and codeword length in the 
finite-state encoder, along with the length of fheslid- 
ing-block decoder window. With regard to this 
measure, the FM code adds only minor complexi- 
ty relative to the uncoded channel: the encoder 
has only one state and can be implemented by 
simply inserting a 1 following each databit; the decoder 
window is two code bits long, and decoding is 
implemented by simply droppingthesecond codebit 
of each detected pair. 

From the trellis structure, we can get a quali- 
tative indicationof the hardware complexity by look- 
ing at the number of states (each corresponding 
to an Add-Compare-Select processor), the num- 
ber of edges per trellis stage (the number of pos- 
sible survivor extensions that need to be examined), 
and the number of samples detected per trellis stage. 
In the FM case, there are only two states, four 
edges per stage, and two samples per stage, again 
representing a minimal incrementin complexity rel- 
ative to the uncoded PRML detector. There is 
even a simple difference metric form for the 
Viterbi algorithm [41], 

As an approach to increasing linear density, 
the advantage of such a coded-PR4 system, 
although possibly substantial, may not be appar- 
ent without careful analysis of the channel signal-to- 
noise ratio as a function of transition density and the 
required channel equalization [42], In disk record- 
ing systems, however, there is a second coordi- 
nate, corresponding to the radial direction, along 
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which density can be increased. Overall area! 
density can beoptimized by choosing the appropriate 
balance of linear and radial (i.e., track) density. 
In this setting, thepotential benefit ofcoded systems 
can be appreciated by considering thefollowinginter- 
esting, although simplistic, argument. 

Supposeonebeginswith a benchmarkPRMLchan- 
nel using a rate 8/9 (0,4/4) code on a nominal track 
width of W, with linear density L, Let's assume that 
head and servo technology permit the physical 
reduction of trackwidthby a factor of three. Onecould 
then dividethe original track into three subtracks, each 
ofwidthW3.Theoryandexperiment[43]indicatethat 
the amplitude of the readback signal andthe noise 
power due to the medium would both be reduced 
by approximately this factor of three, correspond- 
ing to a4.8dBsignal-to-noiscratiopenalty. If, on each 
subtrack, the original channel scheme were applied 
(leading to a tripling of the areal density) the per- 
formance would be unacceptablypoordue to theSNR 
loss. However, if we apply the rate 1/2 interleaved-FM 
code on each subtrack, the gain factorg exactly off- 
sets theSNRloss.implyingthattheprobabililyof error 
on each subtrack will be virtually unchanged from 
thenominal value. The catch, ofcourse, is that the code 
rate on each subtrack is now 1/2, reducing the lin- 
ear density per subtrack to 



(1/2) L 
(8/9) 



The track density has been tripled, however, lead- 
ing to an overall area) density that is a factor of 
27/16 (or approximately 1 .7) times the original, imply- 
ing a 70 percent increase (Fig. 30). 

Although this back-of-the-envelope calcula- 
tion ignores several important technology issues, 
such as intcrtrack interference, narrow-track 
width head design, and position-servo accuracy, it at 
least suggests that the development of practical coded 
systems might provide a route to significant 
increases in areal density in disk drives. 

The example clearly illustrates that the objec- 
tive is to design codes with high rate and large 
coding gain, in order to minimize the track width 
reduction and to provide the greatest noise immu- 
nity and areal density increase in the scenario just 
sketched. Soon we will give examples of codes 
with rate 0/5 andgain factory = 2. Acalculation sim- 
ilar that above shows that cutting the oracfc width 
by two and applying a code with these parameters 
on each subtrack provides an estimated area! 
density increase of almost 80 percent. 

Several methods have been found for the 

■ Figure 30. Track cutting for increased area! 



te 1/2 IB code: 
t areal density: 



costs 4.8 (IB 
gains 4.8 dB 



design ofcoded systems with ACG greater than 
1, for a range of code rates. Thesimplest suchscheme 
isto use an ordinary binary ECCcodeanda precoder 
[32] as shown in Fig. 31. (See also [44].) Assume 
that the ECC code has rate R and minimum Ham- 
ming distance equal to d% in , so that when the 
code is used as an error correction code over a 
random binary channel it is capable of correcting 
{<&< - l)/2 or fewer errors. Recall that an input 0 
to the precoded channel produces an output 0, 
and an input 1 produces an output +1 or -1. It 
follows that the minimal squared-Euclidean distance 
will satisfy the inequality 

4* . 
In fact, it can be shown that for the coded PR4 




system incorporating this code d^ in must be even. 
Therefore, the coded channel has an ACG bound- 
ed below by Rdgjlii d% in is even or K(4L + 
l)/2if<&,isodd. 

To get the full benefit of any coding gain, the 
decoder for this system should be matched both 
to the code and also to the precoded 1 - D 2 par- 
tial-response channel. Just as In the uncoded 
case, one could bit-wise interleave code words 
and treat the channel as two 1 - D channels. If 
the ECC is a convolutional code, the combination 
of the code and the 1 - D channel can be decod- 



combincd Vitcrbi detector/decoder operates on a 
trellis with at most 2 I+I states. 

The coding gain bound suggests that for a 
given rate, one would like to use an ECC with the 
largest minimum Hamming distance. Optimal 
convolutional codes have been foundfor awide range 
of rates and trellis complexity by computer search, 
and tables of these are now available in many text- 
books [45]. Using these codes, the lower bound on 
minimum distance of the trellis-coded PR4 chan- 
nel has been found to be tight in virtually all cases. 

One still has to limit the global runs of consec- 
utive 0s entering the precoder in order to guaran- 
tee a usable signal for timing recovery, as well as 
the runsof Oson the interleaves to reduce path mem- 
ory requirements. One approach to accomplishing 
this is to use a coset of the ECC code, i.e., an 
additive translation of the code sequences, obtained 
by the componentwise addition (modulo 2) of a Fixed 
binary sequence to each code sequence [32]. For a 
rateJt/n code, one could do this by complement- 
ing the code symbols in a specific set of positions 
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Coding gain 3 dB 
• (0,6/1) = (0,44/22) 
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■ Figure 32. Trellis structure for rate 4/5 code with ACG 2.0 




■ Figure 33. Power spectrum of Interleaved-FM and PR4/EPR4 frequency 




■ Figure 34. Power spectrum of EMM and PR1IPR2 frequency response 

in each code word of length n. (Another approach 
might be to first encode the data using a (0, k) 
code, and then to apply a systematic error correct- 
ing code with the desired Hamming distance.) 

An example of the precoded ECC scheme, 
given in [32], makes use of a coset of a rate 4/5 
eonvolutional code, with minimum Hamming dis- 
tance 3, applied to each of the interleaved 1 -D chan- 
nels comprising the PR4 channel. Theresultinggain 
factor isg = 2, providing ACG equal to 2 dB. The 
decoding trellis for each interleave, derived from the 
eight-state trellis of the underlying eonvolutional 
code, has 16 states, as shown in Fig. 32 [32, 33]. 
This codesatisfies the run length constraints (0, Gil) 
= (0,44/22). 

The reader may have observed that the inter- 
leaved-FM code, which has a minimum Hamming 
distance 1 and is not particularly attractive as an ECC 
code, achieves a coding gain that far exceeds the 



lower bound just derived. This code is an exam- 
ple of another class of codes, called matched- 
spectral-nuil (MSN) codes, which recently have been 
shown to provide an efficient method of achiev- 
ing moderate coding gain at high rates. An MSN code 
is designed in such away that the average power spec- 
trum of the wrtte-current waveforms generated 
by the code has the value zero (i.e., a spectral 
null) at frequencieswhere the partial-response chan- 
nel frequency response is zero. 

Aside from the intuitive reasonableness of 
thisdesign criterion (why waste signal power by trans- 
mitting at frequencies where the channel response 
is zero?) it has been shown mathematically that 
matching of the code and channel spectral null 
frequencies provides significant coding gain for a 
large class of partial response channels, including 
those relevant to magnetic and optical recording 
channels [33]. 

For example, Fig. 33 shows the power spec- 
trum of the interleaved-biphase code or, equiva- 
lentiy, the precoded interleaved-FM code, alongside 
the frequency response of the PR4 channel, as 
well as the frequency response of the EFR4 chan- 
nel, where it also provides ACG of 1,8 dB. 

Another example of an MSN code, intended 
for optical recording channels, is the rate 2/3, 
even-mark-modulation (EMM) code [46], It has a 
spectral null at the Nyquist frequency (one-half 
the recorded symbol frequency) and provides 
coding gain factors g = 2 andg = 2.5 for the 1 + 
D (class 1 or PR1) and (1 + D) 2 (class 2 or PR2) 
partial-response channels. The corresponding power 
spectral density and frequency response curves 
are shown in Fig. 34, 

Returning to the 1 - D channel, it hasbeen shown 
[33] that the minimum squared-Euclidean dis- 
tance at the output of the coded channel is bound- 
ed below by 2K, where K is the order of the code's 
spectral null at zero frequency {meaning that thefirst 
2K - 1 derivatives of the code power spectrum 
arc zero at zero frequency). 

Practical codes with spectral nullsof a given order 
at specified frequencies can be designed using the 
sliding-block code construction techniques allud- 
ed to earlier. We remark that the spectral null 
constraints automatically provide the necessary 
rim It ngtl i onstr int ha u< en tie .,t h • . 
i i 1 1 heprcvioussecti I 

trated by the interleaved-biphase code. The ini- 
tial FSTD used in thecode design procedure is chosen 
to describe a family of spectral null sequences 
with capacity large enough to satisfy the target 
coderate. Forexample.Fig. 35 shows such a so-called 
canonical diagram from which one can extract 
initial FSTDs describing sequences with a spec- 
tral null at zero frequency [33], (The labels should 
be interpreted as write-current levels.) 

A rate 8/10 spectral null code with a four-state 
encoder, satisfying (0, G/J) - (0, 10/5) constraints 
when interleaved, was designed for the 1 - D 
channel. Unfortunately, as might be expected, the 
complexity of the encoder finite state machine 
generally increases as the code rate does, so the 
trellis structure reflecting the combination of the 
modulation code and channel can be quite com- 
plicated for high code rates. Indeed, in the case 
of the rate 8/10 MSN code, the trellis would have 
eight states, and 256 branches emanating from 
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As shown in [33], however, there is a natural, 
reduced-complexity Vitcrbi detector for MSN codes 
that asymptotically achieves maximum-likelihood 
performance as a function of the signal-to-noise ratio. 
The detector structure is hased on the much sim- 
pler trellis derived from the initial FSTD used in 
the sliding-block code construction. The MSN 
code sequences belong to the supercede of spec- 
tral null sequences that are represented by the 
reduced-complexity trellis, and the MSN code 
can be designed to ensure that, forany code sequence, 
no sequence generated by the trellis is closer to 
the code sequence (in Euclidean distance) than 
the minimum Euclidean distance of the code. 
Thus, (he decoder can apply the Viterbs algo- 
rithm to the reduced-complexity trellis to find the 
spectral null sequence in the supcrcodc that best fits 
the noisy channel output sequence. In the unlike- 
ly event that the sequence produced by the detec- 
tor is in error or is not in the range of the MSN 
code, the sliding-block decoder limits the propa- 
gation of errors in the decoded data sequence. 
For the rate 8/10 code, the corresponding reduced- 
complexity trellis is shown in Fig. 36. Issues relat- 
ed to the VLSI implementation of an exploratory 
rate 8/10 MSN code for PR4 are discussed in 
more detail in [47]. 

A recently proposed alternative method ofachiev- 
ing coding gain is shown in Fig. 37. Again, a code 
with good Hamming distance is used for the 
ECC code. Now, however, we require that the code 
be chosen from the class of codes that allow for 
:rfiviemdeci dingviaa soft do isiun decoding tlgo- 
rithm. (All convolulional codes fic into this 
class.) A two-step detector is now used. The first 
step uses an enhanced Vitcrbi detector matched 
to the channel itself, but modified so that the 
detector outputs reliability estimates on the bina- 
ry data in the detected data stream. The second step 
invokes a soft decision decoder for the ECC 
code that uses the output of the first Viterbi 
detector as its input. This concatenated detector 
approach, using an extended BCH code with 
code word length 64 and minimum Hamming 
distance six, has been shown to achieve an asymp- 
totic coding gai n of close to 3 dB [48] . 

Other Techniques 

Several otherpromising techniques have been pro- 
posed to increase the density of recording. 
One of these, familiar to communication engi- 
neers, is decision feedback equalization [49, 50]. 
Another is a limited tree search algorithm [5 1] . There 
also is an interesting hybrid technique, using por- 
tions of the peak detection system and the PRML 
approach to detect (l,7)-coded information [52], 
These topics have not been treated here, but the 
reader is encouraged to consult the references for 



Summary 



/n this paper, we have discussed many of the 
types of modulation codes designed for use in stor- 
age devices using magnetic recording. The codes 
are intended to minimize the negative effects of inter- 
symbol interference. Most commercial disk drives 
today use a simple type of detector called a peak detec- 
tor, and a corresponding class of codes called run 



•C300000)' 



■ Figure 35. Canonical diagram for spectral null at zero frequency 
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■ Figure 36. 7;?«it tructure j • rati Si 10 MSN code 




■ Figure 37. A pair of decoders thai use soft deci- 



length-limited (d,k) codes have found wide appli- 
cation. Recently, another recording channel tech- 
nology,basedonsamplingdetection-partial-responsc 
(or PRML), has been introduced in commercial disk 
drives. Thistechnotogyhingesonlheuscof controlled 
intersymbol interference, and it requires a new 
class of codes, called (0, Gil) codes. 

The paper concludedwith several examples iilus- 
tratingthat the introduction of partial response equal- 
ization, sampling detection, and digital signal 
processing has set the stage for the invention and 
application of advanced inoculation and coding tech- 
niques in future storage products. 
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